Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-10 Thread Robert Haas
On Tue, Feb 8, 2011 at 10:54 PM, Joachim Wieland wrote: > On Tue, Feb 8, 2011 at 8:31 PM, Itagaki Takahiro > wrote: >> On Tue, Feb 8, 2011 at 13:34, Robert Haas wrote: >>> So how close are we to having a committable version of this?  Should >>> we push this out to 9.2? >> >> I think so. The feat

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-08 Thread Joachim Wieland
On Tue, Feb 8, 2011 at 8:31 PM, Itagaki Takahiro wrote: > On Tue, Feb 8, 2011 at 13:34, Robert Haas wrote: >> So how close are we to having a committable version of this?  Should >> we push this out to 9.2? > > I think so. The feature is pretty attractive, but more works are required: >  * Re-bas

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-08 Thread Itagaki Takahiro
On Tue, Feb 8, 2011 at 13:34, Robert Haas wrote: > So how close are we to having a committable version of this?  Should > we push this out to 9.2? I think so. The feature is pretty attractive, but more works are required: * Re-base on synchronized snapshots patch * Consider to use pipe also on

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-07 Thread Robert Haas
On Mon, Feb 7, 2011 at 10:42 PM, Joachim Wieland wrote: >> i guess the huge amount of info is showing the patch is just for >> debugging and will be removed before commit, right? > > That's right. So how close are we to having a committable version of this? Should we push this out to 9.2? -- R

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-07 Thread Joachim Wieland
Hi Jaime, thanks for your review! On Sun, Feb 6, 2011 at 2:12 PM, Jaime Casanova wrote: > code review: > > something i found, and is a very simple one, is this warning (there's > a similar issue in _StartMasterParallel with the buf variable) > """ > pg_backup_directory.c: In function ‘_EndMaster

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-06 Thread Jaime Casanova
On Sun, Feb 6, 2011 at 2:12 PM, Jaime Casanova wrote: > On Tue, Feb 1, 2011 at 11:32 PM, Joachim Wieland wrote: >> On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas wrote: >>> The parallel pg_dump portion of this patch (i.e. the still-uncommitted >>> part) no longer applies.  Please rebase. >> >> Her

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-06 Thread Jaime Casanova
On Tue, Feb 1, 2011 at 11:32 PM, Joachim Wieland wrote: > On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas wrote: >> The parallel pg_dump portion of this patch (i.e. the still-uncommitted >> part) no longer applies.  Please rebase. > > Here is a rebased version with some minor changes as well. I have

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-04 Thread Magnus Hagander
On Sat, Feb 5, 2011 at 04:50, Joachim Wieland wrote: > On Thu, Feb 3, 2011 at 11:46 PM, Itagaki Takahiro > wrote: >> It might be better to remove Windows-specific codes from the first try. >> I doubt Windows message queue is the best API in such console-based >> application. I hope we could use t

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-04 Thread Joachim Wieland
On Thu, Feb 3, 2011 at 11:46 PM, Itagaki Takahiro wrote: > I think we have 2 important technical issues here: >  * The consistency is not perfect. Each transaction is started >   with small delays in step 1, but we cannot guarantee no other >   transaction between them. This is exactly where the

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-03 Thread Itagaki Takahiro
On Wed, Feb 2, 2011 at 13:32, Joachim Wieland wrote: > Here is a rebased version with some minor changes as well. I read the patch works as below. Am I understanding correctly? 1. Open all connections in a parent process. 2. Start transactions for each connection in the parent. 3. Spawn chi

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-01 Thread Joachim Wieland
On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas wrote: > The parallel pg_dump portion of this patch (i.e. the still-uncommitted > part) no longer applies.  Please rebase. Here is a rebased version with some minor changes as well. I haven't tested it on Windows now but will do so as soon as the Unix

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-30 Thread Robert Haas
On Wed, Jan 19, 2011 at 12:45 AM, Joachim Wieland wrote: > On Mon, Jan 17, 2011 at 5:38 PM, Jaime Casanova wrote: >> This one is the last version of this patch? if so, commitfest app >> should be updated to reflect that > > Here are the latest patches all of them also rebased to current HEAD. > W

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-23 Thread Heikki Linnakangas
On 21.01.2011 19:11, Euler Taveira de Oliveira wrote: Em 21-01-2011 12:47, Andrew Dunstan escreveu: Maybe we could change the hint to say "--file=DESTINATION" or "--file=FILENAME|DIRNAME" ? ... "--file=OUTPUT" or "--file=OUTPUTNAME". Ok, works for me. I've committed this patch now, with a w

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Euler Taveira de Oliveira
Em 21-01-2011 12:47, Andrew Dunstan escreveu: Maybe we could change the hint to say "--file=DESTINATION" or "--file=FILENAME|DIRNAME" ? ... "--file=OUTPUT" or "--file=OUTPUTNAME". -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@p

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Andrew Dunstan
On 01/21/2011 10:34 AM, Heikki Linnakangas wrote: On 21.01.2011 15:35, Robert Haas wrote: On Fri, Jan 21, 2011 at 4:41 AM, Heikki Linnakangas wrote: There's one UI thing that bothers me. The option to specify the target directory is called --file. But it's clearly not a file. OTOH, I'd hat

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Heikki Linnakangas
On 21.01.2011 15:35, Robert Haas wrote: On Fri, Jan 21, 2011 at 4:41 AM, Heikki Linnakangas wrote: There's one UI thing that bothers me. The option to specify the target directory is called --file. But it's clearly not a file. OTOH, I'd hate to introduce a parallel --dir option just for this.

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Robert Haas
On Fri, Jan 21, 2011 at 4:41 AM, Heikki Linnakangas wrote: > There's one UI thing that bothers me. The option to specify the target > directory is called --file. But it's clearly not a file. OTOH, I'd hate to > introduce a parallel --dir option just for this. Any thoughts on this? If we were star

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Heikki Linnakangas
On 20.01.2011 17:22, Heikki Linnakangas wrote: (I'm working on this, no need to submit a new patch) Ok, here's a heavily refactored version of this (also available at git://git.postgresql.org/git/users/heikki/postgres.git, branch pg_dump_directory). The directory format is now identical to th

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Florian Pflug
On Jan20, 2011, at 16:22 , Heikki Linnakangas wrote: > You can put files in the archive in a certain order if you list them > explicitly in the tar command line, like "tar cf backup.tar toc.dat ...". > It's hard to know the right order, though. In practice you would need to do > "tar tf backup.t

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Heikki Linnakangas
On 20.01.2011 15:46, Joachim Wieland wrote: On Thu, Jan 20, 2011 at 6:07 AM, Heikki Linnakangas wrote: The header is there to identify a file, it contains the header that every other pgdump file contains, including the internal version number and the unique backup id. The tar format doesn't s

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Joachim Wieland
On Thu, Jan 20, 2011 at 6:07 AM, Heikki Linnakangas wrote: >> It's part of the overall idea to make sure files are not inadvertently >> exchanged between different backups and that a file is not truncated. >> In the future I'd also like to add a checksum to the TOC so that a >> backup can be check

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Heikki Linnakangas
On 19.01.2011 16:01, Joachim Wieland wrote: On Wed, Jan 19, 2011 at 7:47 AM, Heikki Linnakangas wrote: Here are the latest patches all of them also rebased to current HEAD. Will update the commitfest app as well. What's the idea of storing the file sizes in the toc file? It looks like it's n

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-19 Thread Joachim Wieland
On Wed, Jan 19, 2011 at 7:47 AM, Heikki Linnakangas wrote: >> Here are the latest patches all of them also rebased to current HEAD. >> Will update the commitfest app as well. > > What's the idea of storing the file sizes in the toc file? It looks like > it's not used for anything. It's part of th

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-19 Thread Heikki Linnakangas
On 19.01.2011 07:45, Joachim Wieland wrote: On Mon, Jan 17, 2011 at 5:38 PM, Jaime Casanova wrote: This one is the last version of this patch? if so, commitfest app should be updated to reflect that Here are the latest patches all of them also rebased to current HEAD. Will update the commitfe

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-17 Thread Jaime Casanova
On Fri, Jan 7, 2011 at 3:18 PM, Joachim Wieland wrote: > Here's a new series of patches for the parallel dump/restore. They need to be > applied on top of each other. > This one is the last version of this patch? if so, commitfest app should be updated to reflect that -- Jaime Casanova