Re: [HACKERS] Multiple --table options for other commands

2012-12-12 Thread Karl O. Pinc
Hi Josh, The good news is that there's only this little bit of documentation formatting to fix On 12/12/2012 11:04:53 PM, Josh Kupershmidt wrote: > On Wed, Dec 12, 2012 at 8:14 AM, Karl O. Pinc wrote: > > On 12/11/2012 10:25:43 PM, Josh Kupershmidt wrote: > >> On Tue, Dec 11, 2012 at 11:46 A

Re: [HACKERS] Multiple --table options for other commands

2012-12-12 Thread Josh Kupershmidt
On Wed, Dec 12, 2012 at 8:14 AM, Karl O. Pinc wrote: > On 12/11/2012 10:25:43 PM, Josh Kupershmidt wrote: >> On Tue, Dec 11, 2012 at 11:46 AM, Karl O. Pinc wrote: >> >> I believe you need ellipses behind --table in the syntax summaries >> >> of the command reference docs. >> >> Hrm, I was followi

Re: [HACKERS] [PERFORM] encouraging index-only scans

2012-12-12 Thread Pavan Deolasee
On Thu, Dec 13, 2012 at 9:21 AM, Tom Lane wrote: > Bruce Momjian writes: >> On Wed, Dec 12, 2012 at 05:27:39PM -0500, Andrew Dunstan wrote: >>> Actually, the table had been analysed but not vacuumed, so this >>> kinda begs the question what will happen to this value on >>> pg_upgrade? Will people

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Jan Wieck
On 12/12/2012 3:12 PM, Simon Riggs wrote: On 12 December 2012 19:13, Jan Wieck wrote: On 12/12/2012 1:12 PM, Simon Riggs wrote: Currently, ANALYZE collects data on all columns and stores these samples in pg_statistic where they can be seen via the view pg_stats. In some cases we have data th

Re: [HACKERS] [PERFORM] encouraging index-only scans

2012-12-12 Thread Tom Lane
Bruce Momjian writes: > On Wed, Dec 12, 2012 at 05:27:39PM -0500, Andrew Dunstan wrote: >> Actually, the table had been analysed but not vacuumed, so this >> kinda begs the question what will happen to this value on >> pg_upgrade? Will people's queries suddenly get slower until >> autovacuum kicks

Re: [HACKERS] [PERFORM] encouraging index-only scans

2012-12-12 Thread Bruce Momjian
On Wed, Dec 12, 2012 at 05:27:39PM -0500, Andrew Dunstan wrote: > > On 12/12/2012 05:12 PM, Andrew Dunstan wrote: > > > >On 12/12/2012 04:32 PM, Tom Lane wrote: > >>Andrew Dunstan writes: > >>>A client is testing a migration from 9.1 to 9.2, and has found that a > >>>large number of queries run m

Re: [HACKERS] logical changeset generation v3 - git repository

2012-12-12 Thread Peter Geoghegan
On 9 December 2012 19:14, Andres Freund wrote: > I pushed a new version which > > - is rebased ontop of master > - is based ontop of the new xlogreader (biggest part) > - is base ontop of the new binaryheap.h > - some fixes > - some more comments I decided to take another look at this, following

Re: [HACKERS] Logical decoding & exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-12 18:52:33 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2012-12-12 12:13:44 +0100, Andres Freund wrote: > >> This morning I wondered whether we couldn't protect against that by > >> acquiring share locks on the catalog rows pg_dump reads, that would > >> result in "could not s

Re: [HACKERS] Logical decoding & exported base snapshot

2012-12-12 Thread Tom Lane
Andres Freund writes: > On 2012-12-12 12:13:44 +0100, Andres Freund wrote: >> This morning I wondered whether we couldn't protect against that by >> acquiring share locks on the catalog rows pg_dump reads, that would >> result in "could not serialize access due to concurrent update" type of >> err

Re: [HACKERS] Logical decoding & exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-12 12:13:44 +0100, Andres Freund wrote: > On 2012-12-11 22:20:18 -0500, Tom Lane wrote: > > Joachim Wieland writes: > > > On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund > > > wrote: > > >> One problem I see is that while exporting a snapshot solves the > > >> visibility issues of the

Re: [HACKERS] Enabling Checksums

2012-12-12 Thread Greg Smith
On 12/5/12 6:49 PM, Simon Riggs wrote: * Zeroing pages, making pages all 1s * Transposing pages * Moving chunks of data sideways in a block * Flipping bits randomly * Flipping data endianness * Destroying particular catalog tables or structures I can take this on, as part of the QA around check

Re: [HACKERS] Use gcc built-in atomic inc/dec in lock.c

2012-12-12 Thread Peter Geoghegan
On 12 December 2012 22:11, Mikko Tiihonen wrote: > noticed a "XXX: It might be worth considering using an atomic fetch-and-add > instruction here, on architectures where that is supported." in lock.c > > Here is my first try at using it. That's interesting, but I have to wonder if there is any ev

[HACKERS] Use gcc built-in atomic inc/dec in lock.c

2012-12-12 Thread Mikko Tiihonen
Hi, I noticed a "XXX: It might be worth considering using an atomic fetch-and-add instruction here, on architectures where that is supported." in lock.c Here is my first try at using it. The patch adds a configure check for gcc 4.7 __atomic_add_fetch as well as the older __sync_add_and_fetch bui

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Tom Lane
Simon Riggs writes: > Currently, ANALYZE collects data on all columns and stores these > samples in pg_statistic where they can be seen via the view pg_stats. Only if you have appropriate privileges. > In some cases we have data that is private and we do not wish others > to see it, such as pati

Re: [HACKERS] Use of systable_beginscan_ordered in event trigger patch

2012-12-12 Thread Dimitri Fontaine
Tom Lane writes: >> Maybe we could just append to it how to remove such an event trigger, >> which is easy to do because you can't target an event trigger related >> command with event triggers, so the following is not disabled: >> DROP EVENT TRIGGER abort_ddl; > > Oh really? The explanation

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Joshua D. Drake
On 12/12/2012 12:12 PM, Simon Riggs wrote: Would protecting it the same way, we protect the passwords in pg_authid, be sufficient? The user backend does need to be able to access the stats data during optimization. It's hard to have data accessible and yet impose limits on the uses to which t

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Simon Riggs
On 12 December 2012 19:13, Jan Wieck wrote: > On 12/12/2012 1:12 PM, Simon Riggs wrote: >> >> Currently, ANALYZE collects data on all columns and stores these >> samples in pg_statistic where they can be seen via the view pg_stats. >> >> In some cases we have data that is private and we do not wis

Re: [HACKERS] [PATCH] PL/Python: Add spidata to all spiexceptions

2012-12-12 Thread Karl O. Pinc
Hi Jan and Oskari, On 12/12/2012 11:36:54 AM, Jan Urbański wrote: > On 10/12/12 19:20, Karl O. Pinc wrote: > > On 12/09/2012 10:33:59 PM, Karl O. Pinc wrote: > > There were 2 outstanding issue raised: > > > > Is this useful enough/proceeding in the right direction to commit > > now? > > I believ

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Jan Wieck
On 12/12/2012 1:12 PM, Simon Riggs wrote: Currently, ANALYZE collects data on all columns and stores these samples in pg_statistic where they can be seen via the view pg_stats. In some cases we have data that is private and we do not wish others to see it, such as patient names. This becomes mor

[HACKERS] PRIVATE columns

2012-12-12 Thread Simon Riggs
Currently, ANALYZE collects data on all columns and stores these samples in pg_statistic where they can be seen via the view pg_stats. In some cases we have data that is private and we do not wish others to see it, such as patient names. This becomes more important when we have row security. Perh

Re: [HACKERS] [PATCH] PL/Python: Add spidata to all spiexceptions

2012-12-12 Thread Jan Urbański
Though not the original patch autor, I did modify and submit it to the CF app, so I'll reply here :) On 10/12/12 19:20, Karl O. Pinc wrote: On 12/09/2012 10:33:59 PM, Karl O. Pinc wrote: I've gone ahead and signed up to review this patch. Thanks! There were 2 outstanding issue raised: I

Re: [HACKERS] PageIsAllVisible()'s trustworthiness in Hot Standby

2012-12-12 Thread Pavan Deolasee
On Wed, Dec 12, 2012 at 1:35 AM, Robert Haas wrote: > On Tue, Dec 4, 2012 at 12:10 PM, Pavan Deolasee > wrote: >> Hmm. Yeah, I do not have guts to prove that either. I'll probably write up a >> comment for your consideration to explain why we don't trust PD_ALL_VISIBLE >> in Hot standby for seq s

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Tom Lane
Heikki Linnakangas writes: > The bug's been fixed now, but note that huge tuples like this will > always cause the table to be extended. Even if there are completely > empty pages in the table, after a vacuum. Even a completely empty > existing page is not considered spacious enough in this cas

Re: [HACKERS] [BUG?] lag of minRecoveryPont in archive recovery

2012-12-12 Thread Fujii Masao
On Wed, Dec 12, 2012 at 2:03 AM, Heikki Linnakangas wrote: > On 11.12.2012 17:04, Fujii Masao wrote: >> >> The patch looks good. I confirmed that the PANIC error didn't happen, >> with the patch. > > > Ok, committed, and also moved the CheckRecoveryConsistency call. Please take > a look to double-

Re: [HACKERS] Multiple --table options for other commands

2012-12-12 Thread Karl O. Pinc
On 12/11/2012 10:25:43 PM, Josh Kupershmidt wrote: > On Tue, Dec 11, 2012 at 11:46 AM, Karl O. Pinc wrote: > > Yes, the current pg_restore silently > > ignores multiple --table arguments, and seems to use the last > > one. You are introducing a backwards incompatible > > change here. > Agreed

Re: [HACKERS] Use of systable_beginscan_ordered in event trigger patch

2012-12-12 Thread Tom Lane
Dimitri Fontaine writes: > Tom Lane writes: >> BTW, is the example in the CREATE EVENT TRIGGER ref page meant as an >> IQ test for unwary readers? > Maybe we could just append to it how to remove such an event trigger, > which is easy to do because you can't target an event trigger related > com

Re: [HACKERS] [v9.3] writable foreign tables

2012-12-12 Thread Albe Laurenz
Erik Rijkers wrote: >> pgsql-v9.3-writable-fdw-poc.v8.part-2.patch 151 k >> pgsql-v9.3-writable-fdw-poc.v8.part-1.patch 70 k > > I wanted to have a look at this, and tried to apply part 1, en then part 2 on > top of that (that's > the idea, right?) > > part 1 applies and then part 2 does not.

Re: [HACKERS] [v9.3] writable foreign tables

2012-12-12 Thread Erik Rijkers
On Wed, December 12, 2012 14:45, Kohei KaiGai wrote: > pgsql-v9.3-writable-fdw-poc.v8.part-2.patch 151 k > pgsql-v9.3-writable-fdw-poc.v8.part-1.patch 70 k I wanted to have a look at this, and tried to apply part 1, en then part 2 on top of that (that's the idea, right?) part 1 applies and t

Re: [HACKERS] Use of systable_beginscan_ordered in event trigger patch

2012-12-12 Thread Dimitri Fontaine
Tom Lane writes: > Yeah, I had forgotten about that loose end, but we definitely need > something of the sort. Committed with additional documentation. > (I put something in the CREATE EVENT TRIGGER ref page, but do we > need anything about it in chapter 37?) Thanks! I guess we could add a note

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread David Gould
On Wed, 12 Dec 2012 14:23:12 +0200 Heikki Linnakangas wrote: > The bug's been fixed now, but note that huge tuples like this will > always cause the table to be extended. Even if there are completely > empty pages in the table, after a vacuum. Even a completely empty > existing page is not co

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Heikki Linnakangas
On 12.12.2012 14:24, David Gould wrote: I don't know if this is the same thing. At least in the comments I was reading trying to figure this out there was some concern that someone else could change the space on the page. Does RelationGetBufferForTuple() guarantee against this too? Yeah, Relati

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread David Gould
On Wed, 12 Dec 2012 13:56:08 +0200 Heikki Linnakangas wrote: > However, RelationGetBufferForTuple() won't return such > a page, it guarantees that the first tuple does indeed fit on the page > it returns. For the same reason, the later check that at least one tuple > was actually placed on the

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Heikki Linnakangas
On 12.12.2012 14:17, David Gould wrote: On Wed, 12 Dec 2012 12:27:11 +0100 Andres Freund wrote: On 2012-12-12 03:04:19 -0800, David Gould wrote: COPY IN loops in heap_multi_insert() extending the table until it fills the Heh. Nice one. Did you hit that in practice? Yeah, with a bunch of

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread David Gould
On Wed, 12 Dec 2012 12:27:11 +0100 Andres Freund wrote: > On 2012-12-12 03:04:19 -0800, David Gould wrote: > > > > COPY IN loops in heap_multi_insert() extending the table until it fills the > Heh. Nice one. Did you hit that in practice? Yeah, with a bunch of hosts that run postgres on a ramdis

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Heikki Linnakangas
On 12.12.2012 13:27, Andres Freund wrote: On 2012-12-12 03:04:19 -0800, David Gould wrote: One more point, in the case where we don't insert any rows, we still do all the dirtying and logging work even though we did not modify the page. I have tried skip all this if no rows are added (nthispage

[HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Andres Freund
On 2012-12-12 03:04:19 -0800, David Gould wrote: > > COPY IN loops in heap_multi_insert() extending the table until it fills the > disk when trying to insert a wide row into a table with a low fill-factor. > Internally fill-factor is implemented by reserving some space space on a > page. For large

Re: [HACKERS] Logical decoding & exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-11 22:39:14 -0500, Steve Singer wrote: > On 12-12-11 06:52 PM, Andres Freund wrote: > >Hi, > > > > >Problem 1: > > > >One problem I see is that while exporting a snapshot solves the > >visibility issues of the table's contents it does not protect against > >schema changes. I am not sure

Re: [HACKERS] Logical decoding & exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-11 22:20:18 -0500, Tom Lane wrote: > Joachim Wieland writes: > > On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund > > wrote: > >> One problem I see is that while exporting a snapshot solves the > >> visibility issues of the table's contents it does not protect against > >> schema change

[HACKERS] bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread David Gould
COPY IN loops in heap_multi_insert() extending the table until it fills the disk when trying to insert a wide row into a table with a low fill-factor. Internally fill-factor is implemented by reserving some space space on a page. For large enough rows and small enough fill-factor bulk_multi_insert

Re: [HACKERS] Logical decoding & exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-11 21:05:51 -0500, Joachim Wieland wrote: > On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund wrote: > > One problem I see is that while exporting a snapshot solves the > > visibility issues of the table's contents it does not protect against > > schema changes. I am not sure whether thats

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-12-12 Thread Amit Kapila
On Wednesday, December 12, 2012 5:23 AM Greg Smith wrote: > On 11/23/12 5:57 AM, Amit kapila wrote: > > Let us try to see by example: > > Total RAM - 22G > > Database size - 16G > >... > > Case -2 (Shared Buffers - 10G) > > a. Load all the files in OS buffers. In best case OS buffers can > contain1

Re: [HACKERS] Timing events WIP v1

2012-12-12 Thread Pavel Stehule
2012/12/12 Greg Smith : > On 11/20/12 8:02 PM, Craig Ringer wrote: >> >> On 11/20/2012 04:48 PM, Pavel Stehule wrote: > > >>> I don't like to see current hstore in core - It doesn't support >>> nesting, it doesn't support different datatypes, it is not well >>> supported by plpgsql >>> >> >> ... or