Re: [HACKERS] Moving ExecInsertIndexTuples and friends to new file

2015-04-23 Thread Heikki Linnakangas
On 04/24/2015 06:30 AM, Stephen Frost wrote: * Peter Geoghegan (p...@heroku.com) wrote: On Thu, Apr 23, 2015 at 12:05 PM, Heikki Linnakangas wrote: While looking at Peter's INSERT ... ON CONFLICT patch, I started to feel that ExecInsertIndexTuples() and friends would deserve a file of their ow

Re: [HACKERS] forward vs backward slashes in msvc build code

2015-04-23 Thread Michael Paquier
On Fri, Mar 13, 2015 at 6:04 PM, Michael Paquier wrote: > On Fri, Mar 13, 2015 at 6:20 AM, Alvaro Herrera wrote: >> Peter Eisentraut wrote: >>> This is contrib/chkpass not finding the crypt symbol, which is >>> presumably in some library. But I can't see how it would normally find >>> it, without

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Kyotaro HORIGUCHI
Hello, At Thu, 23 Apr 2015 14:49:29 -0500, Jim Nasby wrote in <55394cc9.5050...@bluetreble.com> > On 4/23/15 5:07 AM, Kyotaro HORIGUCHI wrote: > > This is because parsing of UNION immediately converts constants > > of unknown type in the UNION's both arms to text so the top level > > select won'

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Amit Kapila
On Fri, Apr 24, 2015 at 1:03 AM, Jim Nasby wrote: > > On 4/23/15 11:45 AM, Petr Jelinek wrote: >> >> On 23/04/15 18:24, Andres Freund wrote: >>> >>> Whether that's feasible complexity wise is debatable, but it's certainly >>> possible. >>> >>> >>> I do wonder what, in realistic cases, is actually

Re: [HACKERS] [committers] pgsql: RLS fixes, new hooks, and new test module

2015-04-23 Thread Stephen Frost
Christian, * Christian Ullrich (ch...@chrullrich.net) wrote: > * Stephen Frost wrote: > > >RLS fixes, new hooks, and new test module > > The buildfarm says that with -DCLOBBER_CACHE_ALWAYS, the RLS > violations get blamed on the wrong tables. Mostly, they are catalogs > (I have seen pg_opclass,

Re: [HACKERS] Moving ExecInsertIndexTuples and friends to new file

2015-04-23 Thread Stephen Frost
* Peter Geoghegan (p...@heroku.com) wrote: > On Thu, Apr 23, 2015 at 12:05 PM, Heikki Linnakangas wrote: > > While looking at Peter's INSERT ... ON CONFLICT patch, I started to feel > > that ExecInsertIndexTuples() and friends would deserve a file of their own, > > and not be buried in the middle

Re: [HACKERS] Fwd: [GENERAL] 4B row limit for CLOB tables

2015-04-23 Thread Tom Lane
Stephen Frost writes: > * Bruce Momjian (br...@momjian.us) wrote: >> On Sun, Feb 1, 2015 at 03:54:03PM +0100, Álvaro Hernández Tortosa wrote: >>> The problem here is that performance degrades exponentially, or >>> worse. Speaking here from experience, we already tested this for a >>> very similar

Re: [HACKERS] a fast bloat measurement tool (was Re: Measuring relation free space)

2015-04-23 Thread Abhijit Menon-Sen
At 2015-04-24 08:35:40 +0530, amit.kapil...@gmail.com wrote: > > > Just stick a PG_RETURN_NULL() at the end? > > That should also work. OK, updated patch attached with just that one change. I'm not doing anything about the rename. I don't know if it's possible to make patch(1) rename a file at a

Re: [HACKERS] a fast bloat measurement tool (was Re: Measuring relation free space)

2015-04-23 Thread Amit Kapila
On Fri, Apr 24, 2015 at 8:04 AM, Abhijit Menon-Sen wrote: > > At 2015-04-24 07:22:27 +0530, amit.kapil...@gmail.com wrote: > > > > Few minor issues: > > 1. > > warning on windows > > > > \pgstatbloat.c(193): warning C4715: 'pgstatbloat' : not all control paths > > return a value > > This is becaus

Re: [HACKERS] Fwd: [GENERAL] 4B row limit for CLOB tables

2015-04-23 Thread Stephen Frost
* Bruce Momjian (br...@momjian.us) wrote: > On Sun, Feb 1, 2015 at 03:54:03PM +0100, Álvaro Hernández Tortosa wrote: > > The problem here is that performance degrades exponentially, or > > worse. Speaking here from experience, we already tested this for a > > very similar case (table creation,

Re: [HACKERS] a fast bloat measurement tool (was Re: Measuring relation free space)

2015-04-23 Thread Abhijit Menon-Sen
At 2015-04-24 07:22:27 +0530, amit.kapil...@gmail.com wrote: > > Few minor issues: > 1. > warning on windows > > \pgstatbloat.c(193): warning C4715: 'pgstatbloat' : not all control paths > return a value This is because the function ends with an ereport(ERROR, …). What would you suggest here? Jus

Re: [HACKERS] Supporting TAP tests with MSVC and Windows

2015-04-23 Thread Michael Paquier
On Mon, Apr 20, 2015 at 9:01 PM, Michael Paquier wrote: > On Sun, Apr 19, 2015 at 10:01 PM, Michael Paquier > wrote: >> Note as well that this patch uses the following patches fixing >> independent issues: >> ... > > Attached is v4. I added a switch in config.pl to be consistent with > ./configur

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Noah Misch
On Thu, Apr 23, 2015 at 10:54:45AM -0400, Tom Lane wrote: > I wrote: > > Given that anole is the only one reporting this, I'm not sure that we > > should immediately blame Postgres itself. I have a vague recollection > > that we've seen this symptom before and traced it to a bug in some > > suppor

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Sawada Masahiko
On Fri, Apr 24, 2015 at 1:31 AM, Jim Nasby wrote: > On 4/23/15 11:06 AM, Petr Jelinek wrote: >> >> On 23/04/15 17:45, Bruce Momjian wrote: >>> >>> On Thu, Apr 23, 2015 at 09:45:38AM -0400, Robert Haas wrote: >>> Agreed, no extra file, and the same write volume as currently. It would >>> also matc

Re: [HACKERS] a fast bloat measurement tool (was Re: Measuring relation free space)

2015-04-23 Thread Amit Kapila
On Wed, Apr 22, 2015 at 6:33 PM, Abhijit Menon-Sen wrote: > > At 2015-04-18 12:28:36 +0530, amit.kapil...@gmail.com wrote: > > > > I think you have missed to address the below point: > > > > >> 4) prefetch > > Updated patch attached, as well as the diff against the earlier version > just to make i

Re: [HACKERS] adding more information about process(es) cpu and memory usage

2015-04-23 Thread Tom Lane
Jeff Janes writes: > On Thu, Apr 23, 2015 at 10:17 AM, Heikki Linnakangas > wrote: >> In a nutshell, I don't think PostgreSQL should get involved in that... > I have often wanted an SQL function which would expose the back-end's > rusage statistics to the front-end. This could support a \timing

Re: [HACKERS] adding more information about process(es) cpu and memory usage

2015-04-23 Thread Jeff Janes
On Thu, Apr 23, 2015 at 10:17 AM, Heikki Linnakangas wrote: > On 04/23/2015 08:00 PM, Radovan Jablonovsky wrote: > >> During current encounters with amazon web services - RDS, the DBA does not >> have access to OS/linux shell of underlying instance. That render some >> postgresql monitoring techn

Re: [HACKERS] [committers] pgsql: RLS fixes, new hooks, and new test module

2015-04-23 Thread Christian Ullrich
* Stephen Frost wrote: RLS fixes, new hooks, and new test module The buildfarm says that with -DCLOBBER_CACHE_ALWAYS, the RLS violations get blamed on the wrong tables. Mostly, they are catalogs (I have seen pg_opclass, pg_am, and pg_amproc), but some also come up with binary garbage instea

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 12:45 PM, Peter Geoghegan wrote: >>> * We need to figure out the tuple lock strength details. I think this >>> is doable, but it is the greatest challenge to committing ON CONFLICT >>> UPDATE at this point. Andres feels that we should require no greater >>> lock strength th

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Michael Paquier
On Thu, Apr 23, 2015 at 10:42 PM, Robert Haas wrote: > On Thu, Apr 23, 2015 at 4:19 AM, Simon Riggs wrote: >> We only need a freeze/backup map for larger relations. So if we map 1000 >> blocks per map page, we skip having a map at all when size < 1000. > > Agreed. We might also want to map multip

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 1:08 PM, Heikki Linnakangas wrote: > That said, I'd actually like to see a separate heap_super_delete() function > for that, rather than piggybacking on heap_delete(). It's a quite different > operation. There'll be some duplication, but seems better than a maze of > if-els

Re: [HACKERS] pg_dump: largeobject behavior issues (possible bug)

2015-04-23 Thread Andrew Dunstan
On 04/23/2015 04:04 PM, Andrew Gierth wrote: "Joshua" == Joshua D Drake writes: Joshua> The database dumps fine as long as we don't dump large Joshua> objects. However, if we try to dump the large objects, FreeBSD Joshua> will kill pg_dump as it will consume all free memory and Joshua>

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread David Steele
On 4/23/15 4:30 PM, Andres Freund wrote: > On 2015-04-23 16:26:09 -0400, Robert Haas wrote: >> But pg_upgrade automates all that, so you can't use pg_upgrade in that >> case. If we add a GUC as I suggested, you can still use pg_upgrade. > > But we also have to live with data directories being in

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Andres Freund
On 2015-04-23 16:26:09 -0400, Robert Haas wrote: > But pg_upgrade automates all that, so you can't use pg_upgrade in that > case. If we add a GUC as I suggested, you can still use pg_upgrade. But we also have to live with data directories being in a shit state forever onward. We won't really be a

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 3:57 PM, Andres Freund wrote: > On 2015-04-23 15:46:20 -0400, Robert Haas wrote: >> Well, we've made backward-incompatible changes before. Not to this >> specific thing, but in general. I don't think there's anything >> preventing us from doing so here, except that we don

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Andres Freund
On 2015-04-23 15:17:55 -0500, Jim Nasby wrote: > Yes, but only after creating a brand new cluster from scratch, which would > then disallow them from putting tablespaces in $PGDATA. pg_dumpall output includes tablespaces. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Jim Nasby
On 4/23/15 11:01 AM, Andres Freund wrote: On April 23, 2015 6:12:05 PM GMT+03:00, Jim Nasby wrote: On 4/22/15 9:41 PM, Bruce Momjian wrote: The case this doesn't catch is referencing a symbolic link that points to the same directory. We can't make it an error so people can use pg_upgrade the

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-23 23:08:34 +0300, Heikki Linnakangas wrote: > The heapam API is not that stable, we've added arguments to those functions > every once in a while, and I don't recall any complaints. I heard some, but not too many, that's true. I know that I've written code that'd be broken/needed even

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 10:53 PM, Andres Freund wrote: On 2015-04-23 12:45:59 -0700, Peter Geoghegan wrote: On Thu, Apr 23, 2015 at 12:55 AM, Andres Freund wrote: I think you misread my statement: I'm saying we don't need the new argument anymore, even if we still do the super-deletion in heap_delete().

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 12:53 PM, Andres Freund wrote: > Unconvinced. Not breaking an API has its worth. Yeah, and I acknowledge that - but it's not something that it's appropriate to encapsulate IMV. Let's just leave it to Heikki...I'd say he has the deciding vote, especially as the reviewer th

Re: [HACKERS] pg_dump: largeobject behavior issues (possible bug)

2015-04-23 Thread Andrew Gierth
> "Joshua" == Joshua D Drake writes: Joshua> The database dumps fine as long as we don't dump large Joshua> objects. However, if we try to dump the large objects, FreeBSD Joshua> will kill pg_dump as it will consume all free memory and Joshua> swap. With Andrew's help we were able to dete

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-23 12:45:59 -0700, Peter Geoghegan wrote: > On Thu, Apr 23, 2015 at 12:55 AM, Andres Freund wrote: > > I think you misread my statement: I'm saying we don't need the new > > argument anymore, even if we still do the super-deletion in > > heap_delete(). Now that the speculative insertion

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Andres Freund
On 2015-04-23 15:46:20 -0400, Robert Haas wrote: > Well, we've made backward-incompatible changes before. Not to this > specific thing, but in general. I don't think there's anything > preventing us from doing so here, except that we don't want to annoy > too many users. I think the number of us

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Jim Nasby
On 4/23/15 5:07 AM, Kyotaro HORIGUCHI wrote: This is because parsing of UNION immediately converts constants of unknown type in the UNION's both arms to text so the top level select won't be bothered by this problem. But the problematic query doesn't have appropriate timing to do that until the f

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 11:00 AM, Bruce Momjian wrote: >> > I have developed the attached patch to warn about creating tablespaces >> > inside the data directory. The case this doesn't catch is referencing a >> > symbolic link that points to the same directory. We can't make it an >> > error so

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 12:55 AM, Andres Freund wrote: > I think you misread my statement: I'm saying we don't need the new > argument anymore, even if we still do the super-deletion in > heap_delete(). Now that the speculative insertion will not be visible > (as in seen on a tuple they could dele

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-23 Thread Andres Freund
On 2015-04-23 15:40:36 -0400, Robert Haas wrote: > The issue is that you have to vacuum a table frequently enough to > avoid accumulating bloat. The frequency with which you need to vacuum > varies depending on the size of the table and how frequently it's > updated. However, a large, heavily-upd

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 10:44 AM, Jim Nasby wrote: > On 4/23/15 8:25 AM, Robert Haas wrote: >> Some users are partitioning tables just so that each >> partition can be autovac'd separately. That really shouldn't be >> required. > > Are they doing this for improved heap scan performance? Index sca

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Jim Nasby
On 4/23/15 11:45 AM, Petr Jelinek wrote: On 23/04/15 18:24, Andres Freund wrote: Whether that's feasible complexity wise is debatable, but it's certainly possible. I do wonder what, in realistic cases, is actually the bigger contributor to the overhead. The tuple header or the padding we liber

[HACKERS] pg_dump: largeobject behavior issues (possible bug)

2015-04-23 Thread Joshua D. Drake
Hello, I have been working a problem with Andrew Gierth (sp?) in regards to pg_dump. Here is the basic breakdown: FreeBSD 10.1 PostgreSQL 9.3.6 64GB ~ memory 500GB database 228G of largeobjects (106M objects) The database dumps fine as long as we don't dump large objects. However, if we try

Re: [HACKERS] Moving ExecInsertIndexTuples and friends to new file

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 12:05 PM, Heikki Linnakangas wrote: > While looking at Peter's INSERT ... ON CONFLICT patch, I started to feel > that ExecInsertIndexTuples() and friends would deserve a file of their own, > and not be buried in the middle of execUtils.c. I propose that we split > execUtils

[HACKERS] Improving vacuum/VM/etc

2015-04-23 Thread Jim Nasby
I mentioned this idea in the "other"[1] vacuum thread [2], but I think it got lost. Kevin Grittner pointed out that there's a potentially huge number of writes we incur over the life of a tuple [3]: (1) WAL log the insert. (2) Write the tuple. (3) Hint and rewrite the tuple. (4) WAL log the f

[HACKERS] Moving ExecInsertIndexTuples and friends to new file

2015-04-23 Thread Heikki Linnakangas
While looking at Peter's INSERT ... ON CONFLICT patch, I started to feel that ExecInsertIndexTuples() and friends would deserve a file of their own, and not be buried in the middle of execUtils.c. I propose that we split execUtils.c into two, moving ExecOpenIndices(), ExecCloseIndices() ExecIns

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Alvaro Herrera
Thanks for posting this. Joshua D. Drake wrote: > First of all I should note that I discussed the approach mentioned above to > pgsql-hackers and got a very interesting comment from Tom Lane that adding > sorting info to Var and TargetEntry nodes was not a very good idea because > it'd break stor

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread David G. Johnston
On Thursday, April 23, 2015, Jeff Davis wrote: > On Thu, Apr 23, 2015 at 1:49 AM, David G. Johnston > > wrote: > > Reading and writing all this I'm convinced you have gotten the idea in > your > > mind an expectation of equivalency and consistency where there really is > > little or none from an

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Jeff Davis
On Thu, Apr 23, 2015 at 1:49 AM, David G. Johnston wrote: > Reading and writing all this I'm convinced you have gotten the idea in your > mind an expectation of equivalency and consistency where there really is > little or none from an overall design perspective. And none insofar as > would merit

Re: [HACKERS] improving speed of make check-world

2015-04-23 Thread Jeff Janes
On Thu, Aug 14, 2014 at 10:45 PM, Peter Eisentraut wrote: > make check-world creates a temporary installation in every subdirectory > it runs a test in, which is stupid: it's very slow and uses a lot of > disk space. It's enough to do this once per run. That is the essence > of what I have impl

Re: [HACKERS] adding more information about process(es) cpu and memory usage

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 08:00 PM, Radovan Jablonovsky wrote: During current encounters with amazon web services - RDS, the DBA does not have access to OS/linux shell of underlying instance. That render some postgresql monitoring technique of process CPU and memory usage, not useful. Even if the AWS provide

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Joshua D. Drake
On 04/23/2015 09:42 AM, Jim Nasby wrote: On 4/23/15 11:24 AM, Andres Freund wrote: I do wonder what, in realistic cases, is actually the bigger contributor to the overhead. The tuple header or the padding we liberally add in many cases... Assuming you're talking about padding between fields.

[HACKERS] adding more information about process(es) cpu and memory usage

2015-04-23 Thread Radovan Jablonovsky
During current encounters with amazon web services - RDS, the DBA does not have access to OS/linux shell of underlying instance. That render some postgresql monitoring technique of process CPU and memory usage, not useful. Even if the AWS provide internal tools/programming interface for monitoring,

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Petr Jelinek
On 23/04/15 18:24, Andres Freund wrote: Whether that's feasible complexity wise is debatable, but it's certainly possible. I do wonder what, in realistic cases, is actually the bigger contributor to the overhead. The tuple header or the padding we liberally add in many cases... The logical o

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Jim Nasby
On 4/23/15 11:24 AM, Andres Freund wrote: I do wonder what, in realistic cases, is actually the bigger contributor to the overhead. The tuple header or the padding we liberally add in many cases... Assuming you're talking about padding between fields... Several years ago Enova paid Command Pro

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Jim Nasby
On 4/23/15 11:06 AM, Petr Jelinek wrote: On 23/04/15 17:45, Bruce Momjian wrote: On Thu, Apr 23, 2015 at 09:45:38AM -0400, Robert Haas wrote: Agreed, no extra file, and the same write volume as currently. It would also match pg_clog, which uses two bits per transaction --- maybe we can reuse so

[HACKERS] Reducing tuple overhead

2015-04-23 Thread Andres Freund
Split into a new thread, the other one is already growing fast enough. This discussion started at http://archives.postgresql.org/message-id/55391469.5010506%40iki.fi On April 23, 2015 6:48:57 PM GMT+03:00, Heikki Linnakangas wrote: >Stop right there. You need to reserve enough space on the page

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Stephen Frost
Robert, On Thursday, April 23, 2015, Robert Haas wrote: > On Thu, Apr 23, 2015 at 6:13 AM, Sandeep Thakkar > > wrote: > > The test case sha2 in contrib pgcrypto module is failing with a diff on > > anole because the results file contains the additional lines as: > > -- > > > > + WARNING: detect

Re: [HACKERS] Fwd: [GENERAL] 4B row limit for CLOB tables

2015-04-23 Thread Bruce Momjian
On Sun, Feb 1, 2015 at 03:54:03PM +0100, Álvaro Hernández Tortosa wrote: > > On 31/01/15 14:55, Roger Pack wrote: > >[...] > > > >Now, the reality is that GetNewOidWithIndex() is going to keep > >incrementing the global OID counter until it finds an OID that isn't in > >the toast table. That mean

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 9:02 AM, Heikki Linnakangas wrote: > That code in ExecWithCheckOptions is not translatable. See style guide: > http://www.postgresql.org/docs/devel/static/nls-programmer.html#NLS-GUIDELINES It's probably going to need to change when I rebase on top of Dean's/Stephen's work

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Petr Jelinek
On 23/04/15 17:45, Bruce Momjian wrote: On Thu, Apr 23, 2015 at 09:45:38AM -0400, Robert Haas wrote: Right. My point is that either you do X 2M times to maintain that fork and the overhead of the file existence, or you do one VACUUM FREEZE. I am saying that 2M is a large number and adding all

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Heikki Linnakangas
On 04/20/2015 07:37 AM, Peter Geoghegan wrote: if (wco->commandType == CMD_INSERT) command = "INSERT-applicable "; else if (wco->commandType == CMD_UPDATE) command = "UPDATE-applicable

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Andres Freund
On April 23, 2015 6:12:05 PM GMT+03:00, Jim Nasby wrote: >On 4/22/15 9:41 PM, Bruce Momjian wrote: >> The case this doesn't catch is referencing a >> symbolic link that points to the same directory. We can't make it an >> error so people can use pg_upgrade these setups. > >Couldn't we make it an

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 06:52:20PM +0300, Heikki Linnakangas wrote: > >Agreed. Are you saying we can't find a way to fit an 8-byte value into > >the existing page in a backward-compatible way? > > I'm sure we can find a way. We've discussed ways to handle page > format updates in pg_upgrade befor

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 06:38 PM, Bruce Momjian wrote: On Thu, Apr 23, 2015 at 10:42:59AM +0300, Heikki Linnakangas wrote: On 04/22/2015 09:24 PM, Robert Haas wrote: I would feel safer if we added a completely new "epoch" counter to the page header, instead of reusing LSNs. But as we all know, changing

Re: [HACKERS] Shouldn't CREATE TABLE LIKE copy the relhasoids property?

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 09:26:50AM -0400, Robert Haas wrote: > >> What I meant was - I didn't see an attachment on that message. > > > > I didn't attach it as people have told me they can just as easily see > > the patch via git, and since it was so similar, I didn't repost it. > > Should I have?

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 06:39 PM, Petr Jelinek wrote: On 23/04/15 17:24, Heikki Linnakangas wrote: On 04/23/2015 05:52 PM, Jim Nasby wrote: I've often wondered if there was some way we could consolidate XMIN/XMAX from multiple tuples at the page level; that could be a big win for OLAP environments where

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 09:45:38AM -0400, Robert Haas wrote: > > Right. My point is that either you do X 2M times to maintain that fork > > and the overhead of the file existence, or you do one VACUUM FREEZE. I > > am saying that 2M is a large number and adding all those X's might > > exceed the

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 06:24:00PM +0300, Heikki Linnakangas wrote: > >I've often wondered if there was some way we could consolidate XMIN/XMAX > >from multiple tuples at the page level; that could be a big win for OLAP > >environments where most of your tuples belong to a pretty small range of > >

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Petr Jelinek
On 23/04/15 17:24, Heikki Linnakangas wrote: On 04/23/2015 05:52 PM, Jim Nasby wrote: On 4/23/15 2:42 AM, Heikki Linnakangas wrote: On 04/22/2015 09:24 PM, Robert Haas wrote: Yeah. We have a serious need to reduce the size of our on-disk format. On a TPC-C-like workload Jan Wieck recently te

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 10:42:59AM +0300, Heikki Linnakangas wrote: > On 04/22/2015 09:24 PM, Robert Haas wrote: > >>I would feel safer if we added a completely new "epoch" counter to the page > >>>header, instead of reusing LSNs. But as we all know, changing the page > >>>format is a problem for i

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 05:02:19PM +0200, Andres Freund wrote: > On 2015-04-23 15:52:40 +0100, Geoff Winkless wrote: > > When I set out I was really only hoping to express a preference as a user; > > on balance I would really rather not have DO IGNORE, if it were possible to > > avoid, because it's

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 05:05:14PM +0200, Andres Freund wrote: > On 2015-04-23 11:00:43 -0400, Bruce Momjian wrote: > > On Thu, Apr 23, 2015 at 09:13:52AM -0400, Robert Haas wrote: > > > I think this is a good thing to do, but I sure wish we could go > > > further and block it completely. That may

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 05:52 PM, Jim Nasby wrote: On 4/23/15 2:42 AM, Heikki Linnakangas wrote: On 04/22/2015 09:24 PM, Robert Haas wrote: Yeah. We have a serious need to reduce the size of our on-disk format. On a TPC-C-like workload Jan Wieck recently tested, our data set was 34% larger than another

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Pavel Stehule
2015-04-23 16:12 GMT+02:00 Robert Haas : > On Thu, Apr 23, 2015 at 9:55 AM, Pavel Stehule > wrote: > >> On Thu, Apr 23, 2015 at 4:56 AM, Pavel Stehule > > >> wrote: > >> > I don't see a contradiction. There is clean agreement, so ERROR level > >> > should > >> > to show the context. NOTICE and W

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Jim Nasby
On 4/22/15 9:41 PM, Bruce Momjian wrote: The case this doesn't catch is referencing a symbolic link that points to the same directory. We can't make it an error so people can use pg_upgrade these setups. Couldn't we make it an ERROR unless IsBinaryUpgrade? -- Jim Nasby, Data Architect, Blue Tr

Re: [HACKERS] Split the 'Server Programming' chapter into two?

2015-04-23 Thread Jim Nasby
On 4/23/15 9:23 AM, Alvaro Herrera wrote: Robert Haas wrote: On Thu, Apr 23, 2015 at 8:21 AM, Andres Freund wrote: To me at least 44 - 47 don't really fit well to the rest. I think we either should invent a new category for them, or move them to 'Internals'. Maybe we could introduce 'Extendi

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Andres Freund
On 2015-04-23 11:00:43 -0400, Bruce Momjian wrote: > On Thu, Apr 23, 2015 at 09:13:52AM -0400, Robert Haas wrote: > > I think this is a good thing to do, but I sure wish we could go > > further and block it completely. That may require more thought than > > we have time to put in at this stage of

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Jim Nasby
On 4/23/15 8:42 AM, Robert Haas wrote: On Thu, Apr 23, 2015 at 4:19 AM, Simon Riggs wrote: We were talking about having an incremental backup map also. Which sounds a lot like the freeze map. Yeah, possibly. I think we should try to set things up so that the backup map can be updated asynchr

Re: [HACKERS] Add CINE for ALTER TABLE ... ADD COLUMN

2015-04-23 Thread Fabrízio de Royes Mello
On Wed, Apr 22, 2015 at 3:48 PM, Payal Singh wrote: > > The following review has been posted through the commitfest application: > make installcheck-world: tested, failed > Implements feature: not tested > Spec compliant: not tested > Documentation:not tested > > Seein

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-23 15:52:40 +0100, Geoff Winkless wrote: > When I set out I was really only hoping to express a preference as a user; > on balance I would really rather not have DO IGNORE, if it were possible to > avoid, because it's really ugly, but DO UPDATE/DO NOTHING I could just > about cope with (

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 09:13:52AM -0400, Robert Haas wrote: > On Wed, Apr 22, 2015 at 10:41 PM, Bruce Momjian wrote: > >> What is a real problem is that we don't block creating tablespaces > >> anywhere at all, including in obviously problematic places like the > >> transaction log directory: > >

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Tom Lane
I wrote: > Given that anole is the only one reporting this, I'm not sure that we > should immediately blame Postgres itself. I have a vague recollection > that we've seen this symptom before and traced it to a bug in some > supporting library. Is anole using any particularly out-of-date versions

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Jim Nasby
On 4/23/15 2:42 AM, Heikki Linnakangas wrote: On 04/22/2015 09:24 PM, Robert Haas wrote: Yeah. We have a serious need to reduce the size of our on-disk format. On a TPC-C-like workload Jan Wieck recently tested, our data set was 34% larger than another database at the beginning of the test, an

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Geoff Winkless
On 23 April 2015 at 14:50, Andres Freund wrote: > > ​Maybe I'm misreading it, but isn't index_predicate meant to be inside > the > > brackets? > > > > > http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/on-conflict-docs/sql-insert.html > > That has changed since. ​Oh, helpful. :)​

Re: [HACKERS] Allow SQL/plpgsql functions to accept record

2015-04-23 Thread Jim Nasby
On 4/23/15 8:33 AM, Pavel Stehule wrote: I agree that that would be useful. I think the problem with an expression like rowvar.something is that PL/pgsql cannot infer the type of the result, and nothing else works without that. I doubt that it's practical to lift that restrictio

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-23 Thread Jim Nasby
On 4/23/15 8:25 AM, Robert Haas wrote: Some users are partitioning tables just so that each partition can be autovac'd separately. That really shouldn't be required. Are they doing this for improved heap scan performance? Index scan performance? If the table wasn't partitioned, would they nee

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Tom Lane
Robert Haas writes: > On Thu, Apr 23, 2015 at 6:13 AM, Sandeep Thakkar > wrote: >> The test case sha2 in contrib pgcrypto module is failing with a diff on >> anole because the results file contains the additional lines as: >> + WARNING: detected write past chunk end in ExprContext 6021cb

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Alvaro Herrera
Robert Haas wrote: > There are lots of machines failing in pg_upgradeCheck, but I don't see > details of the failures in the logs. Yeah, I think the buildfarm script is failing to save the error log. Anyway AFAIR this is related to the move from contrib to src/bin; machines that have updated to b

Re: [HACKERS] Split the 'Server Programming' chapter into two?

2015-04-23 Thread Alvaro Herrera
Robert Haas wrote: > On Thu, Apr 23, 2015 at 8:21 AM, Andres Freund wrote: > > To me at least 44 - 47 don't really fit well to the rest. I think we > > either should invent a new category for them, or move them to > > 'Internals'. Maybe we could introduce 'Extending the Server' category > > for t

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 9:55 AM, Pavel Stehule wrote: >> On Thu, Apr 23, 2015 at 4:56 AM, Pavel Stehule >> wrote: >> > I don't see a contradiction. There is clean agreement, so ERROR level >> > should >> > to show the context. NOTICE and WARNINGs doesn't need it - and there is >> > a >> > backwar

Re: [HACKERS] Split the 'Server Programming' chapter into two?

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 8:21 AM, Andres Freund wrote: > While playing around with where exactly to put the replication > origin/progress docs I once more noticed that the 'Server Programming' > book is a mix of different topics. > > It currently contains: > 35. Extending SQL > 36. Triggers

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 6:13 AM, Sandeep Thakkar wrote: > The test case sha2 in contrib pgcrypto module is failing with a diff on > anole because the results file contains the additional lines as: > -- > > + WARNING: detected write past chunk end in ExprContext 6021cbb0 That's clearly a

Re: [HACKERS] Code paths where LWLock should be released on failure

2015-04-23 Thread Kevin Grittner
Michael Paquier wrote: > I have also been surprised by the inconsistencies particularly in > predicate.c or other places regarding LWLock releases. Sometimes > they are released on failure, sometimes not. Those are not needed for correctness; they are a micro-optimization for SerializableXactHas

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-23 14:34:02 +0100, Geoff Winkless wrote: > > A syntax error. DO is a reserved keyword. Update is just unreserved (and > > thus can be used as a column label). Ignore is unreserved with the patch > > and was unreserved before. We obviously can make both reserved, but of so > > we have to

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Pavel Stehule
2015-04-23 15:47 GMT+02:00 Robert Haas : > On Thu, Apr 23, 2015 at 4:56 AM, Pavel Stehule > wrote: > > I don't see a contradiction. There is clean agreement, so ERROR level > should > > to show the context. NOTICE and WARNINGs doesn't need it - and there is a > > backward compatibility and usabil

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 4:56 AM, Pavel Stehule wrote: > I don't see a contradiction. There is clean agreement, so ERROR level should > to show the context. NOTICE and WARNINGs doesn't need it - and there is a > backward compatibility and usability reasons don't do it. Whether notices and warnings

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Robert Haas
On Wed, Apr 22, 2015 at 8:55 PM, Bruce Momjian wrote: > On Wed, Apr 22, 2015 at 06:36:23PM -0500, Jim Nasby wrote: >> On 4/22/15 6:12 PM, Bruce Momjian wrote: >> >My point is that for the life of 200M transactions, you would have the >> >overhead of an additional file per table in the file system,

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 4:19 AM, Simon Riggs wrote: > We were talking about having an incremental backup map also. Which sounds a > lot like the freeze map. Yeah, possibly. I think we should try to set things up so that the backup map can be updated asynchronously by a background worker, so that

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Geoff Winkless
On 23 April 2015 at 13:51, Andres Freund wrote: > On April 23, 2015 3:34:07 PM GMT+03:00, Geoff Winkless < > pgsqlad...@geoff.dj> wrote: > ​ > ​>​ > And what's to stop me having "SELECT true > ​ > AS >do" in the where clause (as per your UPDATE objection)? > > A syntax error. DO is a reserved k

Re: [HACKERS] Allow SQL/plpgsql functions to accept record

2015-04-23 Thread Pavel Stehule
2015-04-23 15:19 GMT+02:00 Robert Haas : > On Wed, Apr 22, 2015 at 6:12 PM, Jim Nasby > wrote: > > On 4/22/15 2:12 PM, Merlin Moncure wrote: > >> That being said, I think json types with their associated API, given > >> that they are core types, will ultimately handle these types of > >> problems

Re: [HACKERS] Shouldn't CREATE TABLE LIKE copy the relhasoids property?

2015-04-23 Thread Robert Haas
On Wed, Apr 22, 2015 at 8:57 PM, Bruce Momjian wrote: > On Tue, Apr 21, 2015 at 05:36:41PM -0400, Robert Haas wrote: >> On Mon, Apr 20, 2015 at 5:41 PM, Bruce Momjian wrote: >> > On Mon, Apr 20, 2015 at 05:04:14PM -0400, Robert Haas wrote: >> >> On Mon, Apr 20, 2015 at 4:11 PM, Bruce Momjian wro

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-23 Thread Robert Haas
On Wed, Apr 22, 2015 at 5:17 PM, Bruce Momjian wrote: > On Wed, Apr 22, 2015 at 06:07:00PM -0300, Alvaro Herrera wrote: >> > Good point, but doesn't vacuum remove the need for pruning as it removes >> > all the old rows? >> >> Sure. The point, I think, is to make autovacuum runs of some sort that

  1   2   >