Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-12-03 Thread Timothy Garnett
On Tue, Dec 3, 2013 at 12:14 PM, Bruce Momjian wrote: > On Wed, Apr 24, 2013 at 03:33:42PM -0400, Andrew Dunstan wrote: > > > > On 04/23/2013 07:53 PM, Timothy Garnett wrote: > ... > > >Attached is two diffs off of the REL9_2_4 tag that I've been > > >using. The first is a simple change that ser

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-12-03 Thread Bruce Momjian
On Wed, Apr 24, 2013 at 03:33:42PM -0400, Andrew Dunstan wrote: > > On 04/23/2013 07:53 PM, Timothy Garnett wrote: > >Hi All, > > > >Currently the -j option to pg_restore, which allows for > >parallelization in the restore, can only be used if the input file > >is a regular file and not, for ex.,

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-05-16 Thread Timothy Garnett
> If you need something like this short term, we actually found a way to do it > ourselves for a migration we performed back in October. The secret is xargs > with the -P option: > > xargs -I{} -P 8 -a table-list.txt \ > bash -c "pg_dump -Fc -t {} my_db | pg_restore -h remote -d my_db" > > Fill

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-05-16 Thread Shaun Thomas
On 05/16/2013 12:16 PM, Timothy Garnett wrote: I think you'd have to be real careful around foreign key constraints for that to work. Not especially. All you need to do is bootstrap the database with a bunch of empty table targets (no constraints, keys, etc), then restore with the xargs comm

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-25 Thread Shaun Thomas
On 04/25/2013 12:56 PM, Timothy Garnett wrote: As the OP, I'll just note that my organization would definitely find use for a parallel migrator tool as long as it supported doing a selection of tables (i.e. -t / -T) in addition to the whole database and it supported or we were able to patch in a

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-25 Thread Timothy Garnett
On Wed, Apr 24, 2013 at 5:47 PM, Joachim Wieland wrote: > On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner < > ste...@kaltenbrunner.cc> wrote: > >> > What might make sense is something like pg_dump_restore which would have >> > no intermediate storage at all, just pump the data etc from one

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-25 Thread Timothy Garnett
As the OP, I'll just note that my organization would definitely find use for a parallel migrator tool as long as it supported doing a selection of tables (i.e. -t / -T) in addition to the whole database and it supported or we were able to patch in an option to cluster as part of the migration (the

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Claudio Freire
On Wed, Apr 24, 2013 at 6:47 PM, Joachim Wieland wrote: > On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner > wrote: >> >> > What might make sense is something like pg_dump_restore which would have >> > no intermediate storage at all, just pump the data etc from one source >> > to another in

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Joachim Wieland
On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner < ste...@kaltenbrunner.cc> wrote: > > What might make sense is something like pg_dump_restore which would have > > no intermediate storage at all, just pump the data etc from one source > > to another in parallel. But I pity the poor guy who ha

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Stefan Kaltenbrunner
On 04/24/2013 09:51 PM, Andrew Dunstan wrote: > > On 04/24/2013 03:49 PM, Andrew Dunstan wrote: >> >> On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: >>> Andrew Dunstan writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: > Anyways, the question is if people think this is generally use

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/24/2013 03:49 PM, Andrew Dunstan wrote: On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: Andrew Dunstan writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: Anyways, the question is if people think this is generally useful. If so I can clean up the preferred choice a bit and rebase

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: Andrew Dunstan writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: Anyways, the question is if people think this is generally useful. If so I can clean up the preferred choice a bit and rebase it off of master, etc. I find this idea very us

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Dimitri Fontaine
Andrew Dunstan writes: > On 04/23/2013 07:53 PM, Timothy Garnett wrote: >> Anyways, the question is if people think this is generally useful. If so >> I can clean up the preferred choice a bit and rebase it off of master, >> etc. I find this idea very useful yes. Another idea would be to allow

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/23/2013 07:53 PM, Timothy Garnett wrote: Hi All, Currently the -j option to pg_restore, which allows for parallelization in the restore, can only be used if the input file is a regular file and not, for ex., a pipe. However this is a pretty common occurrence for us (usually in the for