Re: [PERFORM] In progress INSERT wrecks plans on table

2013-07-13 Thread Jeff Janes
On Sunday, June 16, 2013, Heikki Linnakangas wrote: > On 06.05.2013 04:51, Mark Kirkwood wrote: > >> On 05/05/13 00:49, Simon Riggs wrote: >> >>> On 3 May 2013 13:41, Simon Riggs wrote: >>> >>> (3) to make the check on TransactionIdIsInProgress() into a heuristic, since we don't *need* to c

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-07-13 Thread Ants Aasma
On Sat, Jul 13, 2013 at 1:47 AM, Jeff Janes wrote: > But I also tried it with 4 pgbench clients, and ran into a collapse of > the performance, TPS dropping down to ~8 TPS. It is too variable to > figure out how reliable the speed-up with this patch is, so far. > Apparently they are all fighting

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-07-12 Thread Jeff Janes
On Mon, May 6, 2013 at 1:14 AM, Simon Riggs wrote: > On 6 May 2013 02:51, Mark Kirkwood wrote: >> On 05/05/13 00:49, Simon Riggs wrote: >>> >>> On 3 May 2013 13:41, Simon Riggs wrote: >>> (3) to make the check on TransactionIdIsInProgress() into a heuristic, since we don't *need* to ch

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-06-16 Thread Simon Riggs
On 16 June 2013 16:23, Heikki Linnakangas wrote: > On 06.05.2013 04:51, Mark Kirkwood wrote: >> >> On 05/05/13 00:49, Simon Riggs wrote: >>> >>> On 3 May 2013 13:41, Simon Riggs wrote: >>> (3) to make the check on TransactionIdIsInProgress() into a heuristic, since we don't *need* to ch

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-06-16 Thread Simon Riggs
On 16 June 2013 16:04, Heikki Linnakangas wrote: > On 03.05.2013 15:41, Simon Riggs wrote: >> >> What appears to be happening is we're spending a lot of time in >> TransactionIdIsInProgress() so we can set hints and then when we find >> it is still in progress we then spend more time in XidIsInSna

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-06-16 Thread Tom Lane
Heikki Linnakangas writes: > I'm not sure if this the same idea as (3) above, but ISTM that > HeapTupleSatisfiesMVCC doesn't actually need to call > TransactionIdIsInProgress(), because it checks XidInMVCCSnapshot(). The > comment at the top of tqual.c says: >> * NOTE: must check TransactionId

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-06-16 Thread Heikki Linnakangas
On 06.05.2013 04:51, Mark Kirkwood wrote: On 05/05/13 00:49, Simon Riggs wrote: On 3 May 2013 13:41, Simon Riggs wrote: (3) to make the check on TransactionIdIsInProgress() into a heuristic, since we don't *need* to check that, so if we keep checking the same xid repeatedly we can reduce the

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-06-16 Thread Heikki Linnakangas
On 03.05.2013 15:41, Simon Riggs wrote: What appears to be happening is we're spending a lot of time in TransactionIdIsInProgress() so we can set hints and then when we find it is still in progress we then spend more time in XidIsInSnapshot() while we check that it is still invisible to us. Even

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-10 Thread Mark Kirkwood
On 11/05/13 01:30, Tom Lane wrote: Mark Kirkwood writes: Unfortunately a trigger will not really do the job - analyze ignores in progress rows (unless they were added by the current transaction), and then the changes made by analyze are not seen by any other sessions. So no changes to plans unt

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-10 Thread Tom Lane
Mark Kirkwood writes: > Unfortunately a trigger will not really do the job - analyze ignores in > progress rows (unless they were added by the current transaction), and > then the changes made by analyze are not seen by any other sessions. So > no changes to plans until the entire INSERT is com

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-10 Thread Mark Kirkwood
(See below for the reply) On 10/05/13 22:48, Vitalii Tymchyshyn wrote: Well, could you write a trigger that would do what you need? AFAIR analyze data is stored no matter transaction boundaries. You could store some counters in session vars and issue an explicit analyze when enough rows were add

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-10 Thread Vitalii Tymchyshyn
Well, could you write a trigger that would do what you need? AFAIR analyze data is stored no matter transaction boundaries. You could store some counters in session vars and issue an explicit analyze when enough rows were added. 7 трав. 2013 08:33, "Mark Kirkwood" напис. > On 07/05/13 18:10, Simo

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-07 Thread Mark Kirkwood
On 07/05/13 19:33, Simon Riggs wrote: On 7 May 2013 07:32, Mark Kirkwood wrote: On 07/05/13 18:10, Simon Riggs wrote: On 7 May 2013 01:23, wrote: I'm thinking that a variant of (2) might be simpler to inplement: (I think Matt C essentially beat me to this suggestion - he originally disco

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-07 Thread Simon Riggs
On 7 May 2013 07:32, Mark Kirkwood wrote: > On 07/05/13 18:10, Simon Riggs wrote: >> >> On 7 May 2013 01:23, wrote: >> >>> I'm thinking that a variant of (2) might be simpler to inplement: >>> >>> (I think Matt C essentially beat me to this suggestion - he originally >>> discovered this issue).

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-07 Thread Matt Clarkson
On Tue, 2013-05-07 at 18:32 +1200, Mark Kirkwood wrote: > On 07/05/13 18:10, Simon Riggs wrote: > > On 7 May 2013 01:23, wrote: > > > >> I'm thinking that a variant of (2) might be simpler to inplement: > >> > >> (I think Matt C essentially beat me to this suggestion - he originally > >> discove

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-06 Thread Mark Kirkwood
On 07/05/13 18:10, Simon Riggs wrote: On 7 May 2013 01:23, wrote: I'm thinking that a variant of (2) might be simpler to inplement: (I think Matt C essentially beat me to this suggestion - he originally discovered this issue). It is probably good enough for only *new* plans to react to the i

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-06 Thread Simon Riggs
On 7 May 2013 01:23, wrote: > I'm thinking that a variant of (2) might be simpler to inplement: > > (I think Matt C essentially beat me to this suggestion - he originally > discovered this issue). It is probably good enough for only *new* plans to > react to the increased/increasing number of in

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-06 Thread mark . kirkwood
> Simon Riggs wrote: > > Patch works and improves things, but we're still swamped by the block > accesses via the index. Which *might* be enough to stop it making the server go unresponsive, we'll look at the effect of this in the next few days, nice work! > > Which brings me back to Mark's origi

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-06 Thread Matt Clarkson
> > (2) seems fairly hard generically, since we'd have to keep track of > the tids returned from the IndexScan to allow us to switch to a > different plan and avoid re-issuing rows that we've already returned. > But maybe if we adapted the IndexScan plan type so that it adopted a > more page orie

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-06 Thread Gavin Flower
On 03/05/13 00:27, Simon Riggs wrote: On 2 May 2013 01:49, Mark Kirkwood wrote: On 02/05/13 02:06, Tom Lane wrote: Mark Kirkwood writes: I am concerned that the deafening lack of any replies to my original message is a result of folk glancing at your original quick reply and thinking... inco

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-06 Thread Simon Riggs
On 6 May 2013 02:51, Mark Kirkwood wrote: > On 05/05/13 00:49, Simon Riggs wrote: >> >> On 3 May 2013 13:41, Simon Riggs wrote: >> >>> (3) to make the check on TransactionIdIsInProgress() into a heuristic, >>> since we don't *need* to check that, so if we keep checking the same >>> xid repeatedly

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-05 Thread Mark Kirkwood
On 05/05/13 00:49, Simon Riggs wrote: On 3 May 2013 13:41, Simon Riggs wrote: (3) to make the check on TransactionIdIsInProgress() into a heuristic, since we don't *need* to check that, so if we keep checking the same xid repeatedly we can reduce the number of checks or avoid xids that seem to

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-04 Thread Simon Riggs
On 3 May 2013 13:41, Simon Riggs wrote: > (3) to make the check on TransactionIdIsInProgress() into a heuristic, > since we don't *need* to check that, so if we keep checking the same > xid repeatedly we can reduce the number of checks or avoid xids that > seem to be long running. That's slightly

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-03 Thread Simon Riggs
On 2 May 2013 23:19, wrote: >> On 2 May 2013 01:49, Mark Kirkwood wrote: >> >> I think we need a problem statement before we attempt a solution, >> which is what Tom is alluding to. >> > > Actually no - I think Tom (quite correctly) was saying that the patch was > not a viable solution. With whi

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-02 Thread mark . kirkwood
> mark.kirkw...@catalyst.net.nz wrote on 03.05.2013 00:19: >> I think the idea of telling postgres that we are doing a load is >> probably >> the wrong way to go about this. We have a framework that tries to >> automatically figure out the best plans...I think some more thought >> about >> how to m

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-02 Thread Thomas Kellerer
mark.kirkw...@catalyst.net.nz wrote on 03.05.2013 00:19: I think the idea of telling postgres that we are doing a load is probably the wrong way to go about this. We have a framework that tries to automatically figure out the best plans...I think some more thought about how to make that understan

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-02 Thread mark . kirkwood
> On 2 May 2013 01:49, Mark Kirkwood wrote: > > I think we need a problem statement before we attempt a solution, > which is what Tom is alluding to. > Actually no - I think Tom (quite correctly) was saying that the patch was not a viable solution. With which I agree. I believe the title of this

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-02 Thread Simon Riggs
On 2 May 2013 01:49, Mark Kirkwood wrote: > On 02/05/13 02:06, Tom Lane wrote: >> >> Mark Kirkwood writes: >>> >>> I am concerned that the deafening lack of any replies to my original >>> message is a result of folk glancing at your original quick reply and >>> thinking... incomplete problem spec

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-01 Thread Mark Kirkwood
On 02/05/13 02:06, Tom Lane wrote: Mark Kirkwood writes: I am concerned that the deafening lack of any replies to my original message is a result of folk glancing at your original quick reply and thinking... incomplete problem spec...ignore... when that is not that case - yes I should have mutt

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-01 Thread Tom Lane
Mark Kirkwood writes: > I am concerned that the deafening lack of any replies to my original > message is a result of folk glancing at your original quick reply and > thinking... incomplete problem spec...ignore... when that is not that > case - yes I should have muttered "9.2" in the original

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-01 Thread Mark Kirkwood
On 26/04/13 15:34, Gavin Flower wrote: On 26/04/13 15:19, Mark Kirkwood wrote: While in general you are quite correct - in the above case (particularly as I've supplied a test case) it should be pretty obvious that any moderately modern version of postgres on any supported platform will exhibit

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-04-25 Thread Mark Kirkwood
On 26/04/13 14:56, Gavin Flower wrote: On 26/04/13 14:33, Mark Kirkwood wrote: Recently we encountered the following unhappy sequence of events: 1/ system running happily 2/ batch load into table begins 3/ very quickly (some) preexisting queries on said table go orders of magnitude slower 4/