Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-13 Thread Andrew Dunstan
On 11/13/2012 10:12 PM, Denis wrote: Please don't think that I'm trying to nitpick here, but pg_dump has options for dumping separate tables and that's not really consistent with the idea that "pg_dump is primarily designed for dumping entire databases". Sure it is. The word "primarily" is

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-13 Thread Jeff Janes
On Tue, Nov 13, 2012 at 7:12 PM, Denis wrote: > Jeff Janes wrote >> On Thu, Nov 8, 2012 at 1:04 AM, Denis < > >> socsam@ > >> > wrote: >>> >>> Still I can't undesrtand why pg_dump has to know about all the tables? >> >> Strictly speaking it probably doesn't need to. But it is primarily >> designe

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-13 Thread Denis
Jeff Janes wrote > On Thu, Nov 8, 2012 at 1:04 AM, Denis < > socsam@ > > wrote: >> >> Still I can't undesrtand why pg_dump has to know about all the tables? > > Strictly speaking it probably doesn't need to. But it is primarily > designed for dumping entire databases, and the efficient way to d

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-09 Thread Jeff Janes
On Thu, Nov 8, 2012 at 1:04 AM, Denis wrote: > > Still I can't undesrtand why pg_dump has to know about all the tables? Strictly speaking it probably doesn't need to. But it is primarily designed for dumping entire databases, and the efficient way to do that is to read it all into memory in a fe

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-08 Thread Denis
Tom Lane-2 wrote > Denis < > socsam@ > > writes: >> Tom Lane-2 wrote >>> Hmmm ... so the problem here isn't that you've got 2600 schemas, it's >>> that you've got 183924 tables. That's going to take some time no matter >>> what. > >> I wonder why pg_dump has to have deal with all these 183924 t

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-07 Thread Tom Lane
Denis writes: > Tom Lane-2 wrote >> Hmmm ... so the problem here isn't that you've got 2600 schemas, it's >> that you've got 183924 tables. That's going to take some time no matter >> what. > I wonder why pg_dump has to have deal with all these 183924 tables, if I > specified to dump only one sc

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-07 Thread Denis
Tom Lane-2 wrote > Denis < > socsam@ > > writes: >> Here is the output of EXPLAIN ANALYZE. It took 5 seconds but usually it >> takes from 10 to 15 seconds when I am doing backup. > >> Sort (cost=853562.04..854020.73 rows=183478 width=219) (actual >> time=5340.477..5405.604 rows=183924 loops=1)

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-06 Thread Tom Lane
Denis writes: > Here is the output of EXPLAIN ANALYZE. It took 5 seconds but usually it > takes from 10 to 15 seconds when I am doing backup. > Sort (cost=853562.04..854020.73 rows=183478 width=219) (actual > time=5340.477..5405.604 rows=183924 loops=1) Hmmm ... so the problem here isn't that y

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-06 Thread Denis
Tom Lane-2 wrote > Denis < > socsam@ > > writes: >> I've read all the posts in thread, and as I understood in version 9.2 >> some >> patches were applied to improve pg_dump speed. I've just installed >> PostgreSQL 9.2.1 and I still have the same problem. I have a database >> with >> 2600 schemas

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-06 Thread Tom Lane
Denis writes: > I've read all the posts in thread, and as I understood in version 9.2 some > patches were applied to improve pg_dump speed. I've just installed > PostgreSQL 9.2.1 and I still have the same problem. I have a database with > 2600 schemas in it. I try to dump each schema individually,

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-11-06 Thread Denis
I've read all the posts in thread, and as I understood in version 9.2 some patches were applied to improve pg_dump speed. I've just installed PostgreSQL 9.2.1 and I still have the same problem. I have a database with 2600 schemas in it. I try to dump each schema individually, but it takes too much

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-09-02 Thread Robert Haas
On Sun, Sep 2, 2012 at 5:39 PM, Jeff Janes wrote: > On Thu, Aug 30, 2012 at 8:17 PM, Tom Lane wrote: >> Robert Haas writes: >>> On Thu, Aug 30, 2012 at 4:51 PM, Tom Lane wrote: Bruce Momjian writes: > On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: >> Ok, I modified

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-09-02 Thread Jeff Janes
On Thu, Aug 30, 2012 at 8:17 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Aug 30, 2012 at 4:51 PM, Tom Lane wrote: >>> Bruce Momjian writes: On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: > Ok, I modified the part of pg_dump where tremendous number of LOCK >

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-08-30 Thread Tom Lane
Robert Haas writes: > On Thu, Aug 30, 2012 at 4:51 PM, Tom Lane wrote: >> Bruce Momjian writes: >>> On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: Ok, I modified the part of pg_dump where tremendous number of LOCK TABLE are issued. I replace them with single LOCK TABLE w

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-08-30 Thread Robert Haas
On Thu, Aug 30, 2012 at 4:51 PM, Tom Lane wrote: > Bruce Momjian writes: >> On Thu, May 31, 2012 at 09:20:43AM +0900, Tatsuo Ishii wrote: Ok, I modified the part of pg_dump where tremendous number of LOCK TABLE are issued. I replace them with single LOCK TABLE with multiple tables.

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Claudio Freire
On Thu, May 31, 2012 at 12:25 PM, Tom Lane wrote: >> No, Tatsuo's patch attacks a phase dominated by latency in some >> setups. > > No, it does not.  The reason it's a win is that it avoids the O(N^2) > behavior in the server.  Whether the bandwidth savings is worth worrying > about cannot be prov

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Tom Lane
Claudio Freire writes: > On Thu, May 31, 2012 at 11:50 AM, Tom Lane wrote: >> The performance patches we applied to pg_dump over the past couple weeks >> were meant to relieve pain in situations where the big server-side >> lossage wasn't the dominant factor in runtime (ie, partial dumps). >> But

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Bruce Momjian
On Thu, May 31, 2012 at 11:04:12AM -0400, Robert Haas wrote: > On Thu, May 31, 2012 at 10:50 AM, Tom Lane wrote: > > Robert Haas writes: > >> On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: > >>> I'm not; Jeff Janes is.  But you shouldn't be holding your breath > >>> anyway, since it's 9.3 mat

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 10:50 AM, Tom Lane wrote: > Robert Haas writes: >> On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: >>> I'm not; Jeff Janes is.  But you shouldn't be holding your breath >>> anyway, since it's 9.3 material at this point. > >> I agree we can't back-patch that change, but

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Claudio Freire
On Thu, May 31, 2012 at 11:50 AM, Tom Lane wrote: > The performance patches we applied to pg_dump over the past couple weeks > were meant to relieve pain in situations where the big server-side > lossage wasn't the dominant factor in runtime (ie, partial dumps). > But this one is targeting exactly

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Bruce Momjian
On Thu, May 31, 2012 at 10:50:51AM -0400, Tom Lane wrote: > Robert Haas writes: > > On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: > >> I'm not; Jeff Janes is. �But you shouldn't be holding your breath > >> anyway, since it's 9.3 material at this point. > > > I agree we can't back-patch that

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Tom Lane
Robert Haas writes: > On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: >> I'm not; Jeff Janes is.  But you shouldn't be holding your breath >> anyway, since it's 9.3 material at this point. > I agree we can't back-patch that change, but then I think we ought to > consider back-patching some var

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: > Claudio Freire writes: >> It's not clear whether Tom is already working on that O(N^2) fix in locking. > > I'm not; Jeff Janes is.  But you shouldn't be holding your breath > anyway, since it's 9.3 material at this point. I agree we can't back-

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Tom Lane
Claudio Freire writes: > It's not clear whether Tom is already working on that O(N^2) fix in locking. I'm not; Jeff Janes is. But you shouldn't be holding your breath anyway, since it's 9.3 material at this point. regards, tom lane -- Sent via pgsql-performance mailing