Re: [HACKERS] pgbench internal contention

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 11:33 AM, Tom Lane wrote: >> Attached is a try at that approach. > > I don't find this to be a particularly safe idea: > >>  #ifndef HAVE_ERAND48 >> -/* we assume all of these are present or missing together */ >> -extern double erand48(unsigned short xseed[3]); >> -extern l

Re: [HACKERS] [RFC] Common object property boards

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 4:27 PM, Alvaro Herrera wrote: > Excerpts from Robert Haas's message of lun ago 01 16:12:56 -0400 2011: >> On Mon, Aug 1, 2011 at 4:02 PM, Alvaro Herrera >> wrote: >> > Excerpts from Kohei KaiGai's message of dom jul 31 02:21:55 -0400 2011: >> >> 2011/7/29 Tom Lane : >> > >

Re: [HACKERS] pgbench internal contention

2011-08-01 Thread Greg Smith
On 07/30/2011 09:08 AM, Robert Haas wrote: If I'm reading the code right, it only modifies __libc_drand48_data on first call, so as long as we called erand48() at least once before spawning the child threads, it would probably work. That seems pretty fragile in the face of the fact that they exp

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread David Fetter
On Mon, Aug 01, 2011 at 09:38:30PM +0100, Dave Page wrote: > On Monday, August 1, 2011, David Fetter wrote: > > On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote: > >> On Mon, Aug 1, 2011 at 6:46 PM, David Fetter wrote: > >> > On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: > >

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Tom Lane
David Fetter writes: > On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote: >> The scope or number of files is irrelevant. Having just a single >> GPL'd file in those installers makes it impossible for ISVs to >> bundle them with their products, unless they open source them under >> a GPL co

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Andrew Dunstan
On 08/01/2011 04:23 PM, David Fetter wrote: The scope or number of files is irrelevant. Having just a single GPL'd file in those installers makes it impossible for ISVs to bundle them with their products, unless they open source them under a GPL compatible licence. I know of no one, not even

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Dave Page
On Monday, August 1, 2011, David Fetter wrote: > On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote: >> On Mon, Aug 1, 2011 at 6:46 PM, David Fetter wrote: >> > On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: >> >> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane wrote: >> >> > I thin

Re: [HACKERS] [RFC] Common object property boards

2011-08-01 Thread Alvaro Herrera
Excerpts from Robert Haas's message of lun ago 01 16:12:56 -0400 2011: > On Mon, Aug 1, 2011 at 4:02 PM, Alvaro Herrera > wrote: > > Excerpts from Kohei KaiGai's message of dom jul 31 02:21:55 -0400 2011: > >> 2011/7/29 Tom Lane : > > > >> > It would likely be better to not expose the struct type,

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread David Fetter
On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote: > On Mon, Aug 1, 2011 at 6:46 PM, David Fetter wrote: > > On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: > >> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane wrote: > >> > I think you had better plan on incorporating GNU readline in

Re: [HACKERS] [RFC] Common object property boards

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 4:02 PM, Alvaro Herrera wrote: > Excerpts from Kohei KaiGai's message of dom jul 31 02:21:55 -0400 2011: >> 2011/7/29 Tom Lane : > >> > It would likely be better to not expose the struct type, just individual >> > lookup functions. >> > >> If so, individual functions to expo

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 7:56 PM, Tom Lane wrote: > However, this means that Dean is not simply adding some self-contained > compression logic; he's eliminating information from the data structure > on the grounds that he can get it from elsewhere.  I think that that > ought to be treated as a sepa

Re: [HACKERS] [RFC] Common object property boards

2011-08-01 Thread Alvaro Herrera
Excerpts from Kohei KaiGai's message of dom jul 31 02:21:55 -0400 2011: > 2011/7/29 Tom Lane : > > It would likely be better to not expose the struct type, just individual > > lookup functions. > > > If so, individual functions to expose a certain property of the supplied > object type should be p

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Dean Rasheed writes: > OK, so I should split this into 2 patches? > Even without the compression, it's probably worth the 16 -> 10 byte > reduction that would result from removing the 2nd CTID in the UPDATE > case, and that part would be a pretty small patch. Yeah, my point exactly. The rest of

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
On 1 August 2011 19:56, Tom Lane wrote: > Dean Rasheed writes: >> On 1 August 2011 18:55, Tom Lane wrote: >>> Robert Haas writes: >> On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed >> wrote: Don't we already do that when pruning HOT chains? > >>> I thought that only happens after the transa

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread Merlin Moncure
On Mon, Aug 1, 2011 at 10:18 AM, Robert Haas wrote: > On Mon, Aug 1, 2011 at 11:04 AM, David Fetter wrote: >> SQL is Turing-complete, so the chances are excellent that it's >> possible to express that algorithm in it. ;) >> >> Look into common table expressions for iteration/recursion, and >> win

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Dean Rasheed writes: > On 1 August 2011 18:55, Tom Lane wrote: >> Robert Haas writes: > On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed wrote: >>> Don't we already do that when pruning HOT chains? >> I thought that only happens after the transaction is committed, and >> old enough, whereas the tr

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
On 1 August 2011 18:55, Tom Lane wrote: > Robert Haas writes: >> On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed >> wrote: Don't we already do that when pruning HOT chains? > >>> I thought that only happens after the transaction is committed, and >>> old enough, whereas the trigger code only

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Andrew Dunstan
On 08/01/2011 02:17 PM, Dave Page wrote: On Mon, Aug 1, 2011 at 6:46 PM, David Fetter wrote: On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane wrote: I think you had better plan on incorporating GNU readline into installer builds for Lion.

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread Dave Page
On Mon, Aug 1, 2011 at 6:46 PM, David Fetter wrote: > On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: >> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane wrote: >> > I think you had better plan on incorporating GNU readline into >> > installer builds for Lion. >> >> Unfortunately the licence

Re: [HACKERS] WIP fix proposal for bug #6123

2011-08-01 Thread Kevin Grittner
"Kevin Grittner" wrote: > By the way, my current patch does break two existing UPDATE > statements in the regression test misc.sql file: Fixed in the attached version of the patch. I consider the trigger behavior addressed by this patch to be a bug, and serious enough to merit inclusion of a

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 6:39 PM, Tom Lane wrote: > Simon Riggs writes: >> One of the things I was looking at doing was allowing the operator >> estimation functions mark the plan as "one-shot" if they used >> non-uniform data to predict the estimate. That would require most >> functions to observe

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Robert Haas writes: > On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed wrote: >>> Don't we already do that when pruning HOT chains? >> I thought that only happens after the transaction is committed, and >> old enough, whereas the trigger code only needs to follow the chain in >> the updating transac

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 1:42 PM, Dean Rasheed wrote: >>> H ... not sure.  It seems a bit scary, but on the other hand we >>> should be able to assume that the updating subtransaction hasn't been >>> rolled back (else surely we shouldn't be firing the trigger).  So in >>> principle it seems like

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-01 Thread David Fetter
On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote: > On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane wrote: > > I think you had better plan on incorporating GNU readline into > > installer builds for Lion. > > Unfortunately the licence makes that a non-starter. What is it about a GPLed psql cl

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
On 1 August 2011 18:36, Robert Haas wrote: > On Mon, Aug 1, 2011 at 1:31 PM, Tom Lane wrote: >> Dean Rasheed writes: >>> On 1 August 2011 17:49, Tom Lane wrote: Ummm ... I only read the data structure comments, not the code, but I don't see where you store the second CTID for an updat

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Tom Lane
Simon Riggs writes: > One of the things I was looking at doing was allowing the operator > estimation functions mark the plan as "one-shot" if they used > non-uniform data to predict the estimate. That would require most > functions to observe the rule that if a plan is marked unsafe then > nobody

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 1:31 PM, Tom Lane wrote: > Dean Rasheed writes: >> On 1 August 2011 17:49, Tom Lane wrote: >>> Ummm ... I only read the data structure comments, not the code, but I >>> don't see where you store the second CTID for an update event? > >> Ah yes, I forgot to mention that bit

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Dean Rasheed writes: > On 1 August 2011 17:49, Tom Lane wrote: >> Ummm ... I only read the data structure comments, not the code, but I >> don't see where you store the second CTID for an update event? > Ah yes, I forgot to mention that bit. I'm using > &(tuple1.t_data->t_ctid) to get the second

Re: [HACKERS] error: could not find pg_class tuple for index 2662

2011-08-01 Thread Tom Lane
daveg writes: > On Sun, Jul 31, 2011 at 11:44:39AM -0400, Tom Lane wrote: >> I think we need to start adding some instrumentation so we can get a >> better handle on what's going on in your database. If I were to send >> you a source-code patch for the server that adds some more logging >> printo

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 4:55 PM, Tom Lane wrote: > Simon Riggs writes: >> On Tue, Jun 14, 2011 at 9:36 PM, Tom Lane wrote: >>> I have already got plans for a significantly more sophisticated approach >>> to this. > >> I'd like to move forwards on this capability in this release cycle. I >> want t

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
On 1 August 2011 17:49, Tom Lane wrote: > Dean Rasheed writes: >> I've been thinking some more about the long-standing problem of the >> AFTER TRIGGER queue using too much memory, and I think that the >> situation can be improved by using some basic compression. > >> Currently each event added to

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Tom Lane
Dean Rasheed writes: > I've been thinking some more about the long-standing problem of the > AFTER TRIGGER queue using too much memory, and I think that the > situation can be improved by using some basic compression. > Currently each event added to the AFTER TRIGGER queue uses 10 bytes > per tri

[HACKERS] PgWest CFP extended for 12 days

2011-08-01 Thread Joshua D. Drake
Hello Hackers! The CFP for West has been extended until the 12th of August. Let's get those talks in! https://www.postgresqlconference.org/talk_types JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development The PostgreSQL C

Re: [HACKERS] lazy vxid locks, v3

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 11:21 AM, Jeff Davis wrote: > On Mon, 2011-08-01 at 08:12 -0400, Robert Haas wrote: >> > Is the "&& LocalTransactionIdIsValid(lxid)" a guard against calling >> > VirtualXactLockTableCleanup twice? Can that happen? Or is it just >> > defensive coding to avoid making an additi

[HACKERS] Compressing the AFTER TRIGGER queue

2011-08-01 Thread Dean Rasheed
I've been thinking some more about the long-standing problem of the AFTER TRIGGER queue using too much memory, and I think that the situation can be improved by using some basic compression. Currently each event added to the AFTER TRIGGER queue uses 10 bytes per trigger per row for INSERTs and DEL

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Tom Lane
Simon Riggs writes: > On Tue, Jun 14, 2011 at 9:36 PM, Tom Lane wrote: >> I have already got plans for a significantly more sophisticated approach >> to this. > I'd like to move forwards on this capability in this release cycle. I > want to be able to tell whether a plan is a one-shot plan, or n

Re: [HACKERS] pgbench internal contention

2011-08-01 Thread Tom Lane
Robert Haas writes: > On Sat, Jul 30, 2011 at 9:08 AM, Robert Haas wrote: >> If I'm reading the code right, it only modifies __libc_drand48_data on >> first call, so as long as we called erand48() at least once before >> spawning the child threads, it would probably work.  That seems pretty >> fr

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 3:34 PM, Heikki Linnakangas wrote: > On 01.08.2011 17:26, Simon Riggs wrote: >> >> On Mon, Aug 1, 2011 at 2:29 PM, Heikki Linnakangas >>  wrote: >>> >>> I believe we code acquire the locks in right order already, and the patch >>> I >>> posted fixes the premature release of

Re: [HACKERS] pgbench internal contention

2011-08-01 Thread Tom Lane
Robert Haas writes: > If I'm reading the code right, it only modifies __libc_drand48_data on > first call, so as long as we called erand48() at least once before > spawning the child threads, it would probably work. That seems pretty > fragile in the face of the fact that they explicitly state th

Re: [HACKERS] lazy vxid locks, v3

2011-08-01 Thread Jeff Davis
On Mon, 2011-08-01 at 08:12 -0400, Robert Haas wrote: > > Is the "&& LocalTransactionIdIsValid(lxid)" a guard against calling > > VirtualXactLockTableCleanup twice? Can that happen? Or is it just > > defensive coding to avoid making an additional assumption? > > lxid there is just a local variable

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread Robert Haas
On Mon, Aug 1, 2011 at 11:04 AM, David Fetter wrote: > SQL is Turing-complete, so the chances are excellent that it's > possible to express that algorithm in it. ;) > > Look into common table expressions for iteration/recursion, and > windowing functions, commonly used for time series. > > http://

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread David Fetter
On Mon, Aug 01, 2011 at 01:23:26PM +0200, Achim Domma wrote: > Am 26.07.2011 um 00:40 schrieb Florian Pflug: > > > On Jul25, 2011, at 22:31 , Achim Domma wrote: > >> Am 25.07.2011 um 14:48 schrieb Florian Pflug: > >>> A more low-level API is provided by > >>> {heap,index}_{beginscan,endscan}, heap

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Heikki Linnakangas
On 01.08.2011 17:26, Simon Riggs wrote: On Mon, Aug 1, 2011 at 2:29 PM, Heikki Linnakangas wrote: I believe we code acquire the locks in right order already, and the patch I posted fixes the premature release of locks at page split. Your patch is good, but it does rely on the idea that we're

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 2:29 PM, Heikki Linnakangas wrote: > On 01.08.2011 14:35, Simon Riggs wrote: >> >> On Mon, Aug 1, 2011 at 11:44 AM, Heikki Linnakangas >>  wrote: >> Does the order of locking of the buffers matter? I'm sure it does. >>> >>> Yep. >> >> Do you mean that the BlockNumbers

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Heikki Linnakangas
On 01.08.2011 14:35, Simon Riggs wrote: On Mon, Aug 1, 2011 at 11:44 AM, Heikki Linnakangas wrote: Does the order of locking of the buffers matter? I'm sure it does. Yep. Do you mean that the BlockNumbers are already in correct sequence, or that you will be adding this code to redo? I j

Re: [HACKERS] lazy vxid locks, v3

2011-08-01 Thread Robert Haas
On Sun, Jul 31, 2011 at 8:32 PM, Jeff Davis wrote: > fpLocalTransactionId is redundant with the lxid, and the explanation is > that one that they have different locking semantics. That looks > reasonable, and it avoided the need for the careful ordering while > starting/ending a transaction that w

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread Florian Pflug
On Aug1, 2011, at 13:23 , Achim Domma wrote: > I have tables which store two integer IDs and a floating point rank. So the > table MyTable might have these columns: > > EntityID -> int > PropertyID -> int > Rank -> float > > My algorithm needs to retrieve EntityID-Rank-Pairs for some given > Pr

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 11:44 AM, Heikki Linnakangas wrote: >> Does the order of locking of the buffers matter? I'm sure it does. > > Yep. Do you mean that the BlockNumbers are already in correct sequence, or that you will be adding this code to redo? --  Simon Riggs   http://ww

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Alexander Korotkov
On Mon, Aug 1, 2011 at 3:25 PM, Simon Riggs wrote: > On Mon, Aug 1, 2011 at 11:47 AM, Thom Brown wrote: > > On 1 August 2011 11:44, Heikki Linnakangas > > wrote: > >> On 01.08.2011 13:13, Simon Riggs wrote: > >>> And what does NSN stand for? :-) > >> > >> Hmm, I don't know actually know what NS

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Thom Brown
On 1 August 2011 12:25, Simon Riggs wrote: > On Mon, Aug 1, 2011 at 11:47 AM, Thom Brown wrote: >> On 1 August 2011 11:44, Heikki Linnakangas >> wrote: >>> On 01.08.2011 13:13, Simon Riggs wrote: And what does NSN stand for? :-) >>> >>> Hmm, I don't know actually know what NSN is an acronym

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 11:47 AM, Thom Brown wrote: > On 1 August 2011 11:44, Heikki Linnakangas > wrote: >> On 01.08.2011 13:13, Simon Riggs wrote: >>> And what does NSN stand for? :-) >> >> Hmm, I don't know actually know what NSN is an acronym for :-). > > Node Sequence Number. Do you have a r

Re: [HACKERS] Access to current database from C-language function

2011-08-01 Thread Achim Domma
Am 26.07.2011 um 00:40 schrieb Florian Pflug: > On Jul25, 2011, at 22:31 , Achim Domma wrote: >> Am 25.07.2011 um 14:48 schrieb Florian Pflug: >>> A more low-level API is provided by {heap,index}_{beginscan,endscan}, >>> heap_{insert,update,delete} and index_insert. However, correct handling of

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Thom Brown
On 1 August 2011 11:44, Heikki Linnakangas wrote: > On 01.08.2011 13:13, Simon Riggs wrote: >> And what does NSN stand for? :-) > > Hmm, I don't know actually know what NSN is an acronym for :-). Node Sequence Number. -- Thom Brown Twitter: @darkixion IRC (freenode): dark_ixion Registered Linux

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Heikki Linnakangas
On 01.08.2011 13:13, Simon Riggs wrote: On Mon, Aug 1, 2011 at 10:38 AM, Heikki Linnakangas wrote: Attached is a patch for that for 9.1/master. The 9.0 GiST replay code was quite different, it will require a separate patch. Hmm, I was assured no changes would be required for Hot Standby for

Re: Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Simon Riggs
On Mon, Aug 1, 2011 at 10:38 AM, Heikki Linnakangas wrote: > On 27.07.2011 17:43, Alexander Korotkov wrote: >> >> OK, thanks. I also found behaviour of GiST(without patch) with streaming >> replication that seems strange for me. On master there are only few >> rightlinks are InvalidBlockNumber whi

Hot standby and GiST page splits (was Re: [HACKERS] WIP: Fast GiST index build)

2011-08-01 Thread Heikki Linnakangas
On 27.07.2011 17:43, Alexander Korotkov wrote: OK, thanks. I also found behaviour of GiST(without patch) with streaming replication that seems strange for me. On master there are only few rightlinks are InvalidBlockNumber while on slave there are a lot of them. I hack gevel for getting index stru

Re: [HACKERS] One-Shot Plans

2011-08-01 Thread Simon Riggs
On Tue, Jun 14, 2011 at 9:36 PM, Tom Lane wrote: > Simon Riggs writes: >> Currently, the planner and executor are mostly independent of each >> other: the planner doesn't really know when the plan will be executed, >> and the executor doesn't know how recently the plan was made. > >> We can work