Re: [PERFORM] Bulk persistence strategy

2017-05-22 Thread Simon Riggs
s for the suggestion though. Perhaps we should look into parameterisable DO statements. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-performance mailing list (pgsql-performance@postgre

Re: [PERFORM] Bulk persistence strategy

2017-05-21 Thread Simon Riggs
ation. That way I can create a Postgres type > that maps from the application object. > > Thanks for the advice. I'll give that a shot. It sounds like you don't know about anonymous code blocks with DO https://www.postgresql.org/docs/devel/static/sql-do.html -- Simon Riggs

Re: [PERFORM] Backup taking long time !!!

2017-01-23 Thread Simon Riggs
invalid backup memes and terminate them. Never fails to surprise me how many people don't read the docs. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-performance mailing list (pgsql-p

Re: [PERFORM] Understanding BRIN index performance

2016-10-03 Thread Simon Riggs
een filtered (and sorted) for the exact data, right? That could be most simply explained if the distribution of your data is not what you think it is. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via p

Re: [PERFORM] Understanding BRIN index performance

2016-10-03 Thread Simon Riggs
ata in dateAdded > actually is sequential and fairly selective (having now() as the default > over a long period of time), this surprises me. > > With a normal btree index, of course, it runs fine: > > https://explain.depesz.com/s/TB5 Btree retains ordering, BRIN does no

Re: [PERFORM] Millions of tables

2016-09-29 Thread Simon Riggs
gt; color &c in a data lake you must accept the possibility of scanning the > entire lake. However, if all fish were in barrels where each barrel had a > particular kind of fish of specific length, size, color &c then the problem > is far simpler. The task of putting the fish in t

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-12-16 Thread Simon Riggs
On 12 December 2014 at 03:31, Simon Riggs wrote: > Also attached is a new parameter called enable_sortedpath which can be > used to turn on/off the sorted path generated by the planner. Now with attachment. (Thanks Jeff!) -- Simon Riggs http://www.2ndQuadra

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-12-11 Thread Simon Riggs
On 12 December 2014 at 03:22, Simon Riggs wrote: > It's a simple patch, but it solves the test cases I know about and > does almost nothing to planning time. Test cases attached. The files marked "pettus_*" are written up from Christophe Pettus' blog. The other

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-12-11 Thread Simon Riggs
On 30 September 2014 at 10:25, Simon Riggs wrote: > On 30 September 2014 00:00, Tom Lane wrote: >> The existing cost estimation >> code effectively assumes that they're perfectly uniformly distributed; >> which is a good average-case assumption but can be horribly

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-12-09 Thread Simon Riggs
On 10 December 2014 at 10:46, Josh Berkus wrote: > On 12/05/2014 08:04 AM, Simon Riggs wrote: >> On 6 December 2014 at 00:45, Merlin Moncure wrote: >> >>> Neat -- got any test cases (would this have prevented OP's problem)? >> >> No test case was posted

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-12-05 Thread Simon Riggs
On 6 December 2014 at 00:45, Merlin Moncure wrote: > Neat -- got any test cases (would this have prevented OP's problem)? No test case was posted, so I am unable to confirm. A test case I produced that appears to be the same issue is fixed. I await confirmation from the OP. -- Sim

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-12-04 Thread Simon Riggs
On 30 September 2014 at 05:53, Simon Riggs wrote: > On 29 September 2014 16:00, Merlin Moncure wrote: >> On Fri, Sep 26, 2014 at 3:06 AM, Simon Riggs wrote: >>> The problem, as I see it, is different. We assume that if there are >>> 100 distinct values and you use

Re: [PERFORM] unnecessary sort in the execution plan when doing group by

2014-11-04 Thread Simon Riggs
On 28 October 2014 06:26, Huang, Suya wrote: >Memory wanted: 3565580K bytes This means "increase work_mem to this value". -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-perform

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-10-02 Thread Simon Riggs
On 1 October 2014 19:56, Josh Berkus wrote: > On 09/30/2014 04:01 PM, Simon Riggs wrote: >> On 30 September 2014 18:28, Jeff Janes wrote: >> >>>> Anyway, in the particular case I posted fixing n_distinct to realistic >>>> numbers (%) fixed the query pl

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-09-30 Thread Simon Riggs
hn retired 15 years ago and hasn't logged on since, while Johannes > was hired yesterday and never logged on before then. Ah, OK, essentially the same example. Which is why I ruled out correlation stats based approaches and suggested a risk-weighted cost approach. -- Simon R

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-09-30 Thread Simon Riggs
On 30 September 2014 00:00, Tom Lane wrote: > Simon Riggs writes: >> The way I'm seeing it, you can't assume the LIMIT will apply to any >> IndexScan that doesn't have an index condition. If it has just a >> filter, or nothing at all, just an ordering then it

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-09-30 Thread Simon Riggs
On 29 September 2014 22:54, Josh Berkus wrote: > On 09/26/2014 01:06 AM, Simon Riggs wrote: >> On 23 September 2014 00:56, Josh Berkus wrote: >> >>> We've hashed that out a bit, but frankly I think it's much more >>> profitable to pursue fixing the act

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-09-29 Thread Simon Riggs
On 29 September 2014 16:00, Merlin Moncure wrote: > On Fri, Sep 26, 2014 at 3:06 AM, Simon Riggs wrote: >> The problem, as I see it, is different. We assume that if there are >> 100 distinct values and you use LIMIT 1 that you would only need to >> scan 1% of rows. We as

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-09-26 Thread Simon Riggs
able in a very homogenous layout. When data is not, and it seldom is, we get problems. Simply put, assuming that LIMIT will reduce the size of all scans is just way wrong. I've seen many plans where increasing the LIMIT dramatically improves the plan. If we can at least agree

Re: [PERFORM] Postgres Replaying WAL slowly

2014-09-17 Thread Simon Riggs
e ideas. Patch implements option 2 in the above. Skipping the locks entirely seems like it opens a can of worms. Skipping the lock for temp tables is valid since locks don't need to exist on the standby. Any catalog entries for them will exist, but the rows will show them as temp and

Re: [PERFORM] Postgres Replaying WAL slowly

2014-07-03 Thread Simon Riggs
this to the list. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-06-16 Thread Simon Riggs
On 16 June 2013 16:23, Heikki Linnakangas wrote: > On 06.05.2013 04:51, Mark Kirkwood wrote: >> >> On 05/05/13 00:49, Simon Riggs wrote: >>> >>> On 3 May 2013 13:41, Simon Riggs wrote: >>> >>>> (3) to make the check on TransactionIdIsInProg

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-06-16 Thread Simon Riggs
On 16 June 2013 16:04, Heikki Linnakangas wrote: > On 03.05.2013 15:41, Simon Riggs wrote: >> >> What appears to be happening is we're spending a lot of time in >> TransactionIdIsInProgress() so we can set hints and then when we find >> it is still in pro

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-07 Thread Simon Riggs
On 7 May 2013 07:32, Mark Kirkwood wrote: > On 07/05/13 18:10, Simon Riggs wrote: >> >> On 7 May 2013 01:23, wrote: >> >>> I'm thinking that a variant of (2) might be simpler to inplement: >>> >>> (I think Matt C essentially beat me to this

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-06 Thread Simon Riggs
sent. Clearly one would need to avoid doing this too > often (!) - possibly only when number of changed rows > > autovacuum_analyze_scale_factor proportion of the relation concerned or > similar. Are you loading using COPY? Why not break down the load into chunks? -- Simon Riggs

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-06 Thread Simon Riggs
On 6 May 2013 02:51, Mark Kirkwood wrote: > On 05/05/13 00:49, Simon Riggs wrote: >> >> On 3 May 2013 13:41, Simon Riggs wrote: >> >>> (3) to make the check on TransactionIdIsInProgress() into a heuristic, >>> since we don't *need* to check t

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-04 Thread Simon Riggs
On 3 May 2013 13:41, Simon Riggs wrote: > (3) to make the check on TransactionIdIsInProgress() into a heuristic, > since we don't *need* to check that, so if we keep checking the same > xid repeatedly we can reduce the number of checks or avoid xids that > seem to be lon

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-03 Thread Simon Riggs
ly we can reduce the number of checks or avoid xids that seem to be long running. That's slightly more coding than my quick hack here but seems worth it. I think we need both (1) and (3) but the attached patch does just (1). This is a similar optimisation to the one I introduced for Tra

Re: [PERFORM] "WHERE 1 = 2 OR ..." makes planner choose a very inefficient plan

2013-05-02 Thread Simon Riggs
t be a good > answer either, because it would penalize well-written queries to benefit > badly-written ones.) The situation shown could be the result of SQL injection attack. It would be nice to have a switch to do additional checks on SQL queries to ensure such injections don't cause long

Re: [PERFORM] Query planner ignoring constraints on partitioned tables when joining

2013-05-02 Thread Simon Riggs
he join and use that as a constraint in the scan on the second table. We rely on that mechanism for nested loop joins, so we could do with that here also. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via p

Re: [PERFORM] In progress INSERT wrecks plans on table

2013-05-02 Thread Simon Riggs
act with some ease. The plan is set using stats that are set when there are very few non-NULL rows, and those increase massively on load. The way to cope is to run the ANALYZE immediately after the load and then don't allow auto-ANALYZE to reset them later. -- Simon Riggs

Re: [PERFORM] Performance die when COPYing to table with bigint PK

2011-08-01 Thread Simon Riggs
lyze and all other things > - nothing helped, only drop helps. > > Is this known and expected behavior? This is a duplicate post with one on BUGS, being discussed there. --  Simon Riggs   http://www.2ndQuadrant.com/  PostgreSQL Development, 24x7 Support, Training &

Re: [PERFORM] Postgres 9.0.4 + Hot Standby + FusionIO Drive + Performance => Query failed ERROR: catalog is missing 1 attribute(s) for relid 172226

2011-05-08 Thread Simon Riggs
you only get this error once? --  Simon Riggs   http://www.2ndQuadrant.com/  PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Group commit and commit delay/siblings

2010-12-08 Thread Simon Riggs
its with no > scanning of ProcArray if the minimum is 0, and allows setting the > siblings to 0 to enable that shortcut: Minor patch, no downsides. Docs checked. Committed. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services

Re: [PERFORM] partition pruning

2010-03-08 Thread Simon Riggs
o the plan can't take into account > the value of current_timestamp in forming the plan. It could, but it doesn't yet. Partition removal can take place in the executor and this is currently targeted for 9.1. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-performance mailing l

Re: [PERFORM] Air-traffic benchmark

2010-02-04 Thread Simon Riggs
lity, in my experience because it implies the queries you're running aren't ad-hoc. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Calculation of unused columns

2009-10-19 Thread Simon Riggs
On Mon, 2009-10-19 at 13:58 -0400, Tom Lane wrote: > > Most read-only functions are stable or even immutable. Huh? I mean a function that only contains SELECTs. (How would those ever be Stable or Immutable??) -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-perfo

Re: [PERFORM] Calculation of unused columns

2009-10-19 Thread Simon Riggs
On Mon, 2009-10-19 at 13:43 -0400, Tom Lane wrote: > Simon Riggs writes: > > On Sat, 2009-10-17 at 21:41 -0400, Tom Lane wrote: > >> one thing we'd have to consider > >> is whether it is okay to suppress calculation of columns containing > >> volatile f

Re: [PERFORM] Calculation of unused columns

2009-10-19 Thread Simon Riggs
iously). That would allow us to optimize such calls away, if appropriate. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Speeding up a query.

2009-07-07 Thread Simon Riggs
n 10 minute boundaries, rounding up their timeslot requirement. (The single odd timeslot appointment will always waste 1 timeslot). Hope that helps. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performa

Re: [PERFORM] Pointers needed on optimizing slow SQL statements

2009-06-06 Thread Simon Riggs
The Function Index solution works, but it would be much better if we could get the planner to remember certain selectivities. I'm thinking a command like ANALYZE foo [WHERE .... ] which would specifically analyze the selectivity of the given WHERE clause for use in queries. --

Re: [PERFORM] Any better plan for this query?..

2009-05-20 Thread Simon Riggs
On Wed, 2009-05-20 at 07:17 -0400, Robert Haas wrote: > On Wed, May 20, 2009 at 4:11 AM, Simon Riggs wrote: > > The Hash node is fully executed before we start pulling rows through the > > Hash Join node. So the Hash Join node will know at execution time > > whether or no

Re: [PERFORM] Any better plan for this query?..

2009-05-20 Thread Simon Riggs
an just ask the Hash Join at execution time whether it should perform a sort or just pass rows through (act as a no-op). The cost of the Sort node can either be zero, or pro-rated down from the normal cost based upon what we think the probability is of going multi-batch, which would vary by work_m

Re: [PERFORM] Any better plan for this query?..

2009-05-19 Thread Simon Riggs
t; > where both "id" columns are UNIQUE with an index. Do we eliminate > "table2.id" from the ORDER BY in this case? Yes, that is eliminated via equivalence classes. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pg

Re: [PERFORM] Any better plan for this query?..

2009-05-19 Thread Simon Riggs
On Tue, 2009-05-19 at 08:58 -0400, Tom Lane wrote: > Simon Riggs writes: > > Both plans for this query show an IndexScan on a two column-index, with > > an Index Condition of equality on the leading column. The ORDER BY > > specifies a sort by the second index column, so

Re: [PERFORM] Any better plan for this query?..

2009-05-19 Thread Simon Riggs
ce workload growing to 16 sessions it's jumping to 2.5ms, > then with 32 sessions it's 18ms, etc.. Is it just bad all the time, or does it get worse over time? Do you get the same behaviour as 32 sessions if you run 16 sessions for twice as long? -- Simon Riggs www.2

Re: [PERFORM] Any better plan for this query?..

2009-05-19 Thread Simon Riggs
On Tue, 2009-05-19 at 12:36 +0100, Simon Riggs wrote: > Partially sorted data takes much less effort to sort (OK, not zero, I > grant) so this seems like a high complexity, lower value feature. I > agree it should be on the TODO, just IMHO at a lower priority than some > other featur

Re: [PERFORM] Any better plan for this query?..

2009-05-19 Thread Simon Riggs
s an executor feature. Partially sorted data takes much less effort to sort (OK, not zero, I grant) so this seems like a high complexity, lower value feature. I agree it should be on the TODO, just IMHO at a lower priority than some other features. -- Simon Riggs www.2ndQuadr

Re: [PERFORM] Any better plan for this query?..

2009-05-19 Thread Simon Riggs
On Mon, 2009-05-18 at 19:00 -0400, Tom Lane wrote: > Simon Riggs writes: > > In particular, running the tests repeatedly using > > H.REF_OBJECT = '01' > > rather than varying the value seems likely to benefit MySQL. One thing to note in terms of opt

Re: [PERFORM] Any better plan for this query?..

2009-05-19 Thread Simon Riggs
am surprised we are mis-estimating the selectivity. > I think yes (but of course I did not try to replay it several times) If you could that would be appreciated. We don't want to go chasing after something that is not repeatable. -- Simon Riggs www.2ndQuadrant.com Postgre

Re: [PERFORM] Any better plan for this query?..

2009-05-18 Thread Simon Riggs
a scalability wall for hash joins. Is that a repeatable issue? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Any better plan for this query?..

2009-05-18 Thread Simon Riggs
EADME.html >http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/preprepare/preprepare/ Looks very cool Dimitri -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to

Re: [PERFORM] Any better plan for this query?..

2009-05-18 Thread Simon Riggs
numbers on 32+ cores. Try shared_buffer_partitions = 256 -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Any better plan for this query?..

2009-05-14 Thread Simon Riggs
u be re-running the Read-Only tests? Can you run the Dtrace script to assess LWlock contention during the run? Would you re-run the tests with a patch? Thanks, -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Simon Riggs
ugs we'd hit...) If we did, who would care? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Simon Riggs
ncreasing numbers of connections on GetSnapshotData(). Most discussion has been on -hackers, not -perform. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make change

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Simon Riggs
On Tue, 2009-05-12 at 16:00 +0100, Matthew Wakeling wrote: > won't connect operations be all handled by a > single thread - the parent postmaster? No, we spawn then authenticate. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sen

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Simon Riggs
n on the WALWriteLock. Increase wal_buffers to about x8 where you have it now. You can move pg_xlog to its own set of drives. Set checkpoint_completion_target to 0.95. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing li

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Simon Riggs
tard :-)) idiot - because betting while missing > knowledge, and bastard - because knowing the truth is not honset to > get a profit from idiots :-)) That's why I never betting in my life, > but every time telling the same story in such situation... Did you > like it? ;-)) No, but

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Simon Riggs
results then reasonable observers might be persuaded you did not wish to show PostgreSQL in its best light. You up for it? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To ma

Re: [PERFORM] Any better plan for this query?..

2009-05-11 Thread Simon Riggs
ffort control, rather than the more arcane *_limit knobs which are effectively the same thing, just harder to use in practice. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) T

Re: [PERFORM] Any better plan for this query?..

2009-05-11 Thread Simon Riggs
On Mon, 2009-05-11 at 17:18 +0200, Dimitri wrote: > Yes, forget, MySQL is reaching 17.500 TPS here. Please share your measurements of MySQL scalability also. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing l

Re: [PERFORM] Any better plan for this query?..

2009-05-07 Thread Simon Riggs
ility issue - there is one on 8.3.7, because on 32 cores > with such kind of load it's using only 50% CPU and not outpassing > 6.000 TPS, while 8.4 uses 90% CPU and reaching 11.000 TPS.. Yeh, small changes make a big difference. Thanks for the info. How does MySQL perform? -- Simon R

Re: [PERFORM] Any better plan for this query?..

2009-05-07 Thread Simon Riggs
re is a scalability issue here at all. Postgres can clearly do the same query in about the same time. We just have a case where MySQL happens to optimise it well and Postgres doesn't. Since we can trivially design cases that show the opposite I'm not worried too much. -- Simon Riggs

Re: [PERFORM] Transparent table partitioning in future version of PG?

2009-05-07 Thread Simon Riggs
ut saying it again and again is the best way to get this across. You've highlighted that partitioning is a feature with many underlying requirements: infrequent access to data (frequently historical), striping for parallelism and getting around RDBMS flaws (if any). We must be careful t

Re: [PERFORM] Transparent table partitioning in future version of PG?

2009-05-06 Thread Simon Riggs
On Wed, 2009-05-06 at 17:55 -0400, Tom Lane wrote: > Simon Riggs writes: > > On Fri, 2009-05-01 at 11:27 -0400, Robert Haas wrote: > >> The problem has been finding someone who has both the time and the > >> ability to do the work. > > > Unfortunately th

Re: [PERFORM] Any better plan for this query?..

2009-05-06 Thread Simon Riggs
a plan that on this occasion is actually better that way is in no way representative of general performance. Does MySQL support hash joins? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@post

Re: [PERFORM] Transparent table partitioning in future version of PG?

2009-05-06 Thread Simon Riggs
later patches adding about another 10-20 days work. I'd estimate the required work as 60-100 days work from primary author, plus planning and discussion time. YMMV. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing

Re: [PERFORM] performance for high-volume log insertion

2009-04-22 Thread Simon Riggs
educe network roundtrips to a centralised log server. Preparing statements might show up well on tests because people will do tests against a local database, most likely. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance ma

Re: [PERFORM] error updating a very large table

2009-04-15 Thread Simon Riggs
f possible on OS) Otherwise we'll always be wondering what caused the error. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Question on pgbench output

2009-04-05 Thread Simon Riggs
o include the much higher than normal data transfer into your measurements and pgbench won't help there. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to yo

Re: [PERFORM] plpgsql arrays

2009-04-03 Thread Simon Riggs
a common enough problem that having a non-standard join algorithm written for that case would be interesting indeed. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To ma

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-21 Thread Simon Riggs
for people to come up with new strategies that seem to make sense and then we'll test them all. Trying too hard to arrive at the best strategy purely through discussion will mean we miss a few tricks. Feels like we're on the right track here. -- Simon Riggs www.2ndQuadrant.com P

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Simon Riggs
throw it away wholesale and allocate a > new queue. It avoids a whole lot of queue manipulation. Yes, that sounds good. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support - Sent via pgsql-performance mailing list (pgsql-performance@postgresql.or

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Simon Riggs
On Wed, 2009-03-18 at 16:26 -0400, Tom Lane wrote: > Simon Riggs writes: > > On Mon, 2009-03-16 at 16:26 +, Matthew Wakeling wrote: > >> One possibility would be for the locks to alternate between exclusive > >> and > >> shared - that is: > >>

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Simon Riggs
On Wed, 2009-03-18 at 11:45 +, Matthew Wakeling wrote: > On Wed, 18 Mar 2009, Simon Riggs wrote: > > I agree with that, apart from the "granting no more" bit. > > > > The most useful behaviour is just to have two modes: > > * exclusive-lock held - all

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Simon Riggs
y we queue up exclusive locks, but there is no need to since for ProcArrayLock commits are all changing different data. The most useful behaviour is just to have two modes: * exclusive-lock held - all other x locks welcome, s locks queue * shared-lock held - all other s locks welcome, x locks queue Th

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Simon Riggs
ALWriteLock) blocks insertion of new WAL data, > and it would be nice to preserve that property. Yeh, that's just what we'd discussed previously: http://markmail.org/message/gectqy3yzvjs2hru#query:Reworking%20WAL% 20locking+page:1+mid:gectqy3yzvjs2hru+state:results Are you thinking of d

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-17 Thread Simon Riggs
On Tue, 2009-03-17 at 19:54 -0400, Jignesh K. Shah wrote: > > Simon Riggs wrote: > > On Tue, 2009-03-17 at 17:41 -0400, Jignesh K. Shah wrote: > > > > > >> I did a quick test with patch. Unfortunately it improves my number > >> even with default setti

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-17 Thread Simon Riggs
ogic is same, implementation is slightly different for default > set) OK, I bite. 25% gain from doing nothing??? You're stretching my... err, credulity. I like the train of thought for setting 1 and it is worth investigating, but something feels wrong somewhere. -- Simon Riggs

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-17 Thread Simon Riggs
On Wed, 2009-03-11 at 22:20 -0400, Jignesh K. Shah wrote: > A tunable does not impact existing behavior Why not put the tunable parameter into the patch and then show the test results with it in? If there is no overhead, we should then be able to see that. -- Simon Riggs

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-14 Thread Simon Riggs
x27;d be interested in knowing the number of S and X locks requested, so we can think about this from first principles. My understanding is that ratio of S:X is about 10:1. Do you have more exact numbers? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent v

Re: [PERFORM] caching written values?

2009-01-22 Thread Simon Riggs
but not by any directly supported mechanism. You have to consider whether the data you saved would still be required when the server restarts. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@post

Re: [PERFORM] dbt-2 tuning results with postgresql-8.3.5

2008-12-24 Thread Simon Riggs
that the > more checkpoint segments you have, the more time the database spends > fsyncing when at a checkpoint? I would ignore the checkpoint_segment tests because you aren't using a realistic value of shared_buffers. I doubt any such effect is noticeable when you use a realistic value

Re: [PERFORM] Backup strategies

2008-10-17 Thread Simon Riggs
that we must rollforward from. There is no freezing. > Why > not run with pg_start_backup() always enabled? It's not a mode that can be enabled/disabled. Its a starting point. You should run pg_start_backup() each time you run a backup, just like the fine manual describes. Chec

Re: [PERFORM] Choosing a filesystem

2008-09-23 Thread Simon Riggs
ween them at all. The delays we introduced are all in the write phase. Whether that is important or not depends upon OS parameter settings. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postg

Re: [PERFORM] Intel's X25-M SSD

2008-09-22 Thread Simon Riggs
On Mon, 2008-09-08 at 19:12 -0400, Greg Smith wrote: > If like me you've been reading all the flash SSD drive reviews... Great post, thanks for the information. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance

Re: [HACKERS] [PERFORM] autovacuum: use case for indenpedent TOAST table autovac settings

2008-08-14 Thread Simon Riggs
'd want to treat toast tables differently with regard to autovacuuming. It's one more setting to get wrong, so no thanks. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Outer joins and equivalence

2008-06-02 Thread Simon Riggs
On Mon, 2008-06-02 at 18:10 +0100, Simon Riggs wrote: > On Tue, 2008-05-27 at 17:43 -0400, Tom Lane wrote: > > Simon Riggs <[EMAIL PROTECTED]> writes: > > > I have a complex query where making a small change to the SQL increases > > > run-time by > 1000 times

Re: [PERFORM] Outer joins and equivalence

2008-06-02 Thread Simon Riggs
On Tue, 2008-05-27 at 17:43 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > I have a complex query where making a small change to the SQL increases > > run-time by > 1000 times. > > > The first SQL statement is of the form > > >

Re: [PERFORM] 2GB or not 2GB

2008-06-01 Thread Simon Riggs
your view of serious I suppose. I would say it is an acceptable situation, but needs further optimization. I threw some ideas around on Hackers around Dec/New Year, but I don't have time to work on this further myself in this dev cycle. Further contributions welcome. -- Simon Riggs

Re: [PERFORM] 2GB or not 2GB

2008-05-31 Thread Simon Riggs
his improved in 8.3? There is an optimum for each specific sort. Your results cannot be used to make a global recommendation about the setting of work_mem. So not finding any benefit in your tests *and* Magnus seeing an improvement are not inconsistent events. -- Simon Riggs www.2nd

Re: [PERFORM] GEQO Benchmark

2008-05-28 Thread Simon Riggs
put; sometimes it gets the right plan, other times it doesn't get close. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.p

Re: [PERFORM] Outer joins and equivalence

2008-05-28 Thread Simon Riggs
On Wed, 2008-05-28 at 11:45 +0100, Matthew Wakeling wrote: > On Tue, 27 May 2008, Simon Riggs wrote: > > I do recognise that we would *not* be able to deduce this form of SQL > > > > A JOIN B ON (a.id = c.id) LEFT JOIN C ON (b.id = c.id) > > Surely that would not b

Re: [PERFORM] I/O on select count(*)

2008-05-27 Thread Simon Riggs
> > > Or even better: set the hint-bits, but don't dirty the page. > > Which in fact is what Simon suggested, not the other thing. Just raised this on -hackers, BTW. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via

[PERFORM] Outer joins and equivalence

2008-05-27 Thread Simon Riggs
ON (b.id = c.id) though that restriction on outer join equivalence is not relevant here. (SQL, EXPLAINs etc available off-list only, by request). I'm looking into this more now. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-per

Re: [PERFORM] I/O on select count(*)

2008-05-16 Thread Simon Riggs
the transactionid is assigned the page of the clog that contains that transactionid is checked to see if it already exists and if not, it is initialised. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-pe

Re: [PERFORM] Join runs for > 10 hours and then fills up >1.3TB of disk space

2008-05-16 Thread Simon Riggs
e an estimated 3 billion rows? Can you cope if that estimate is wrong and the true figure is much higher? Do you think the estimate is realistic? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-performance mailing list (pgsql-perfor

Re: [PERFORM] Backup causing poor performance - suggestions

2008-05-05 Thread Simon Riggs
reports about Skype Tools, which has both a Slony-like replicator > (not as configurable as Slony, but easier to set up and use), plus an > entirely separate set of scripts that simplifies "warm standby" using > WAL logging. I think we should mention Warm Standby via pg_standby, w

Re: [PERFORM] SQL Function Slowness, 8.3.0

2008-04-17 Thread Simon Riggs
On Thu, 2008-04-17 at 12:41 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > On Thu, 2008-04-17 at 12:12 -0400, Tom Lane wrote: > >> Aren't these suggestions mutually contradictory? > > > No, they're orthogonal. The pretty printer

  1   2   3   4   >