Re: [HACKERS] Parallel pg_dump for 9.1

2010-04-01 Thread Stefan Kaltenbrunner
Jeff wrote: On Mar 30, 2010, at 8:15 AM, Stefan Kaltenbrunner wrote: Peter Eisentraut wrote: On tis, 2010-03-30 at 08:39 +0200, Stefan Kaltenbrunner wrote: on fast systems pg_dump is completely CPU bottlenecked Might be useful to profile why that is. I don't think pg_dump has historically

Re: [HACKERS] Parallel pg_dump for 9.1

2010-04-01 Thread Jeff
On Mar 30, 2010, at 8:15 AM, Stefan Kaltenbrunner wrote: Peter Eisentraut wrote: On tis, 2010-03-30 at 08:39 +0200, Stefan Kaltenbrunner wrote: on fast systems pg_dump is completely CPU bottlenecked Might be useful to profile why that is. I don't think pg_dump has historically been develope

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-30 Thread Stefan Kaltenbrunner
Peter Eisentraut wrote: On tis, 2010-03-30 at 08:39 +0200, Stefan Kaltenbrunner wrote: on fast systems pg_dump is completely CPU bottlenecked Might be useful to profile why that is. I don't think pg_dump has historically been developed with CPU efficiency in mind. It's not pg_dump that is t

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-30 Thread Pierre C
On Tue, 30 Mar 2010 13:01:54 +0200, Peter Eisentraut wrote: On tis, 2010-03-30 at 08:39 +0200, Stefan Kaltenbrunner wrote: on fast systems pg_dump is completely CPU bottlenecked Might be useful to profile why that is. I don't think pg_dump has historically been developed with CPU efficien

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-30 Thread Peter Eisentraut
On tis, 2010-03-30 at 08:39 +0200, Stefan Kaltenbrunner wrote: > on fast systems pg_dump is completely CPU bottlenecked Might be useful to profile why that is. I don't think pg_dump has historically been developed with CPU efficiency in mind. -- Sent via pgsql-hackers mailing list (pgsql-hack

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Stefan Kaltenbrunner
Tom Lane wrote: Josh Berkus writes: On 3/29/10 7:46 AM, Joachim Wieland wrote: I actually assume that whenever people are interested in a very fast dump, it is because they are doing some maintenance task (like migrating to a different server) that involves pg_dump. In these cases, they would

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Andrew Dunstan
Robert Haas wrote: It's completely possible that you could want to clone a server for dev and have more CPU and I/O bandwidth available than can be efficiently used by a non-parallel pg_dump. But certainly what Joachim is talking about will be a good start. I think there is merit to the sync

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Robert Haas
On Mon, Mar 29, 2010 at 4:11 PM, Tom Lane wrote: > Josh Berkus writes: >> On 3/29/10 7:46 AM, Joachim Wieland wrote: >>> I actually assume that whenever people are interested >>> in a very fast dump, it is because they are doing some maintenance >>> task (like migrating to a different server) tha

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Tom Lane
Josh Berkus writes: > On 3/29/10 7:46 AM, Joachim Wieland wrote: >> I actually assume that whenever people are interested >> in a very fast dump, it is because they are doing some maintenance >> task (like migrating to a different server) that involves pg_dump. In >> these cases, they would stop t

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Josh Berkus
On 3/29/10 7:46 AM, Joachim Wieland wrote: > I actually assume that whenever people are interested > in a very fast dump, it is because they are doing some maintenance > task (like migrating to a different server) that involves pg_dump. In > these cases, they would stop their system anyway. Actual

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Robert Haas
On Mon, Mar 29, 2010 at 1:16 PM, Stefan Kaltenbrunner wrote: > Robert Haas wrote: >> >> On Mon, Mar 29, 2010 at 10:46 AM, Joachim Wieland wrote: > > [...] >>> >>> - Regarding the output of pg_dump I am proposing two solutions. The >>> first one is to introduce a new archive type "directory" where

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Stefan Kaltenbrunner
Robert Haas wrote: On Mon, Mar 29, 2010 at 10:46 AM, Joachim Wieland wrote: [...] - Regarding the output of pg_dump I am proposing two solutions. The first one is to introduce a new archive type "directory" where each table and each blob is a file in a directory, similar to the experimental "f

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Robert Haas
On Mon, Mar 29, 2010 at 10:46 AM, Joachim Wieland wrote: > - There are ideas on how to solve the issue with the consistent > snapshot but in the end you can always solve it by stopping your > application(s). I actually assume that whenever people are interested > in a very fast dump, it is because

Re: [HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread David Fetter
On Mon, Mar 29, 2010 at 04:46:48PM +0200, Joachim Wieland wrote: > People have been talking about a parallel version of pg_dump a few > times already. I have been working on some proof-of-concept code for > this feature every now and then and I am planning to contribute this > for 9.1. > > There a

[HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Joachim Wieland
People have been talking about a parallel version of pg_dump a few times already. I have been working on some proof-of-concept code for this feature every now and then and I am planning to contribute this for 9.1. There are two main issues with a parallel version of pg_dump: The first one is that