Re: [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tatsuo Ishii
> I'm not excited by this patch. It dodges the O(N^2) lock behavior for > the initial phase of acquiring the locks, but it does nothing for the > lock-related slowdown occurring in all pg_dump's subsequent commands. > I think we really need to get in the server-side fix that Jeff Janes is > workin

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tom Lane
Stephen Frost writes: > The current situation where the client-to-server latency accounts for > multiple minutes of time is just ridiculous, however, so I feel we need > some form of this patch, even if the server side is magically made much > faster. The constant back-and-forth isn't cheap. No,

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Tatsuo Ishii writes: > > Shall I commit to master and all supported branches? > > I'm not excited by this patch. It dodges the O(N^2) lock behavior for > the initial phase of acquiring the locks, but it does nothing for the > lock-related slowdown occurri

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tom Lane
Tatsuo Ishii writes: >> Ok, I modified the part of pg_dump where tremendous number of LOCK >> TABLE are issued. I replace them with single LOCK TABLE with multiple >> tables. With 100k tables LOCK statements took 13 minutes in total, now >> it only takes 3 seconds. Comments? > Shall I commit to m

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tatsuo Ishii
>> Yeah, Jeff's experiments indicated that the remaining bottleneck is lock >> management in the server. What I fixed so far on the pg_dump side >> should be enough to let partial dumps run at reasonable speed even if >> the whole database contains many tables. But if psql is taking >> AccessShar

Re: [PERFORM] does the query planner consider work_mem?

2012-05-30 Thread Murat Tasan
Ants -- you're on the right track: i tried your suggestion and found that at times during subsequent executions the performance will drop down to about 6 seconds. though periodically it jumps back up to about 3 minutes, and there's no other DB server usage during these times (i.e. i'm the only one

Re: [PERFORM] does the query planner consider work_mem?

2012-05-30 Thread Ants Aasma
On Wed, May 30, 2012 at 8:57 PM, Murat Tasan wrote: > any insights here? Have you tried running the slow option multiple times? According to the explain output all of the time was accounted to the bitmap heap scan. For the second explain plan the same node was fast. It looks to me as the first ex

[PERFORM] does the query planner consider work_mem?

2012-05-30 Thread Murat Tasan
hi all - i'm having a bit of trouble with some queries that are running painfully slowly after migrating my database from one machine using PostgreSQL 8.2 to another machine with PostgreSQL 8.4. as far as i can tell, the two *servers* (not the physical machines) are set up pretty close to identical

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tatsuo Ishii
> Yeah, Jeff's experiments indicated that the remaining bottleneck is lock > management in the server. What I fixed so far on the pg_dump side > should be enough to let partial dumps run at reasonable speed even if > the whole database contains many tables. But if psql is taking > AccessShareLock