Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-08-30 Thread Tom Lane
Robert Haas writes: > On Thu, Aug 30, 2012 at 4:51 PM, Tom Lane wrote: >> Bruce Momjian writes: >>> On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: Ok, I modified the part of pg_dump where tremendous number of LOCK TABLE are issued. I replace them with single LOCK TABLE w

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-08-30 Thread Robert Haas
On Thu, Aug 30, 2012 at 4:51 PM, Tom Lane wrote: > Bruce Momjian writes: >> On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: Ok, I modified the part of pg_dump where tremendous number of LOCK TABLE are issued. I replace them with single LOCK TABLE with multiple tables.

Re: [PERFORM] pg_dump and thousands of schemas

2012-08-30 Thread Tom Lane
Bruce Momjian writes: > On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: >>> Ok, I modified the part of pg_dump where tremendous number of LOCK >>> TABLE are issued. I replace them with single LOCK TABLE with multiple >>> tables. With 100k tables LOCK statements took 13 minutes in tot

Re: [PERFORM] pg_dump and thousands of schemas

2012-08-30 Thread Bruce Momjian
On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: > >> Yeah, Jeff's experiments indicated that the remaining bottleneck is lock > >> management in the server. What I fixed so far on the pg_dump side > >> should be enough to let partial dumps run at reasonable speed even if > >> the who

Re: [PERFORM] pg_dump and thousands of schemas

2012-08-30 Thread Bruce Momjian
On Thu, Aug 30, 2012 at 04:51:56PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: > >>> Ok, I modified the part of pg_dump where tremendous number of LOCK > >>> TABLE are issued. I replace them with single LOCK TABLE with multiple >

Re: [PERFORM] Question about caching on full table scans

2012-08-30 Thread Scott Marlowe
On Thu, Aug 30, 2012 at 11:34 AM, Markus Innerebner wrote: > Hi Laurenz, > > > In your approach 1 to 3, what do you mean with "load into main memory"? > > > > I forgot to say: I use Java and connect with JDBC. > > in approach 1 I do an initial loading of the entire relation, by executing 1 > SQL q

Re: [PERFORM] Question about caching on full table scans

2012-08-30 Thread Jeff Janes
On Thu, Aug 30, 2012 at 10:34 AM, Markus Innerebner wrote: > > > To flush the filesystem cache (from Linux 2.6.16 on), use > > sync; echo 3 > /proc/sys/vm/drop_caches > > > I started to do that , and > yes, this solves my problem!! > > I assume that deleting file system cache implies that also pos

Re: [PERFORM] Question about caching on full table scans

2012-08-30 Thread Markus Innerebner
Hi Laurenz, > > In your approach 1 to 3, what do you mean with "load into main memory"? I forgot to say: I use Java and connect with JDBC. in approach 1 I do an initial loading of the entire relation, by executing 1 SQL query to load all edges in main memory, where I create my main memory st

Re: [PERFORM] Question about caching on full table scans

2012-08-30 Thread Albe Laurenz
Markus Innerebner wrote: > I am doing some runtime experiments in my implementation, which is computing multi-modal range queries > for a query point (if you want to know details check the website: www.isochrones.inf.unibz.it). > The network is explored using Dijkstra Shortest Path algorithm that s

Re: [PERFORM] Investigating the reason for a very big TOAST table size

2012-08-30 Thread Daniel Farina
On Thu, Aug 30, 2012 at 1:34 AM, Liron Shiri wrote: > We do not use in-database operators to modify the toasted data. > The update operations we perform on the problematic table are in the form of > > UPDATE foo SET field='value' WHERE nid = to_uid(#objId#) Ah, well, there goes that idea, althoug

Re: [PERFORM] Investigating the reason for a very big TOAST table size

2012-08-30 Thread Liron Shiri
We do not use in-database operators to modify the toasted data. The update operations we perform on the problematic table are in the form of UPDATE foo SET field='value' WHERE nid = to_uid(#objId#) -Original Message- From: Daniel Farina [mailto:dan...@heroku.com] Sent: Thursday, August

[PERFORM] Question about caching on full table scans

2012-08-30 Thread Markus Innerebner
Hello PG Performance group, I am doing some runtime experiments in my implementation, which is computing multi-modal range queries for a query point (if you want to know details check the website: www.isochrones.inf.unibz.it). The network is explored using Dijkstra Shortest Path algorithm that s

Re: [PERFORM] Investigating the reason for a very big TOAST table size

2012-08-30 Thread Daniel Farina
On Mon, Aug 27, 2012 at 11:24 PM, Liron Shiri wrote: > There were no "hot standby" configuration, but the DB has start grow fast > after restoring from a base backup as described in > http://www.postgresql.org/docs/8.3/static/continuous-archiving.html#BACKUP-BASE-BACKUP I'm trying to confirm a