Re: [HACKERS] Include WAL in base backup

2011-01-30 Thread Robert Haas
On Sun, Jan 30, 2011 at 9:34 PM, Fujii Masao wrote: > On Mon, Jan 31, 2011 at 4:22 AM, Magnus Hagander wrote: >>> In pg_basebackup.sgml >>>     --checkpoint >> class="parameter">fast|spread >>> >>> Though this is not the problem of the patch, I found the inconsistency >>> of the descriptions abou

Re: [HACKERS] Include WAL in base backup

2011-01-30 Thread Fujii Masao
On Mon, Jan 31, 2011 at 4:22 AM, Magnus Hagander wrote: >> In pg_basebackup.sgml >>     --checkpoint > class="parameter">fast|spread >> >> Though this is not the problem of the patch, I found the inconsistency >> of the descriptions about options of pg_basebackup. We should >> s/--checkpoint/--che

Re: [HACKERS] Include WAL in base backup

2011-01-30 Thread Magnus Hagander
On Thu, Jan 27, 2011 at 05:44, Fujii Masao wrote: > On Wed, Jan 26, 2011 at 5:17 AM, Magnus Hagander wrote: >> We should, and the easiest way is to actually use XLogRead() since the >> code is already there. How about the way in this patch? > > Thanks for the update. I reread the patch. > > +    

Re: [HACKERS] Include WAL in base backup

2011-01-30 Thread Heikki Linnakangas
On 29.01.2011 09:10, Fujii Masao wrote: On Sat, Jan 29, 2011 at 6:02 AM, Heikki Linnakangas wrote: On 27.01.2011 06:44, Fujii Masao wrote: + XLByteToSeg(endptr, endlogid, endlogseg); + /* Have we reached our stop position yet? */ + i

Re: [HACKERS] Include WAL in base backup

2011-01-28 Thread Fujii Masao
On Sat, Jan 29, 2011 at 6:02 AM, Heikki Linnakangas wrote: > On 27.01.2011 06:44, Fujii Masao wrote: >> >> +               XLByteToSeg(endptr, endlogid, endlogseg); >> >> +                       /* Have we reached our stop position yet? */ >> +                       if (logid>  endlogid || >> +  

Re: [HACKERS] Include WAL in base backup

2011-01-28 Thread Heikki Linnakangas
On 27.01.2011 06:44, Fujii Masao wrote: + XLByteToSeg(endptr, endlogid, endlogseg); + /* Have we reached our stop position yet? */ + if (logid> endlogid || + (logid == endlogid&& logseg>= endlogseg)) +

Re: [HACKERS] Include WAL in base backup

2011-01-26 Thread Fujii Masao
On Wed, Jan 26, 2011 at 5:17 AM, Magnus Hagander wrote: > We should, and the easiest way is to actually use XLogRead() since the > code is already there. How about the way in this patch? Thanks for the update. I reread the patch. + MemSet(&statbuf, 0, sizeof(statbuf)); +

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Magnus Hagander
On Tue, Jan 25, 2011 at 16:34, Magnus Hagander wrote: > On Tue, Jan 25, 2011 at 15:04, Fujii Masao wrote: >> On Tue, Jan 25, 2011 at 10:28 PM, Fujii Masao wrote: (the discussed changse above have been applied and pushed to github) >>> >>> Thanks! I'll test and review that. >> >> WAL file mi

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Magnus Hagander
On Tue, Jan 25, 2011 at 15:04, Fujii Masao wrote: > On Tue, Jan 25, 2011 at 10:28 PM, Fujii Masao wrote: >>> (the discussed changse above have been applied and pushed to github) >> >> Thanks! I'll test and review that. > > WAL file might get recycled or removed while walsender is reading it. > So

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Fujii Masao
On Tue, Jan 25, 2011 at 10:28 PM, Fujii Masao wrote: >> (the discussed changse above have been applied and pushed to github) > > Thanks! I'll test and review that. WAL file might get recycled or removed while walsender is reading it. So the WAL file which pg_basebackup seemingly received successf

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Fujii Masao
On Tue, Jan 25, 2011 at 8:21 PM, Magnus Hagander wrote: >> +               elog(DEBUG1, "Going to send wal from %i.%i to %i.%i", >> +                        logid, logseg, >> +                        endlogid, endlogseg); >> >> %u should be used instead of %i. Or how about logging the filename? >

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Magnus Hagander
On Tue, Jan 25, 2011 at 10:56, Fujii Masao wrote: > On Tue, Jan 25, 2011 at 12:33 AM, Magnus Hagander wrote: >> Here's an updated version of the patch: >> >> * rebased on the current git master (including changing the switch >> from -w to -x since -w was used now) >> * includes some documentation

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Fujii Masao
On Tue, Jan 25, 2011 at 12:33 AM, Magnus Hagander wrote: > Here's an updated version of the patch: > > * rebased on the current git master (including changing the switch > from -w to -x since -w was used now) > * includes some documentation > * use XLogByteToSeg and uint32 as mentioned here > * re

Re: [HACKERS] Include WAL in base backup

2011-01-24 Thread Magnus Hagander
On Mon, Jan 24, 2011 at 09:03, Fujii Masao wrote: > On Mon, Jan 24, 2011 at 4:47 PM, Magnus Hagander wrote: >> On Mon, Jan 24, 2011 at 08:45, Fujii Masao wrote: >>> On Fri, Jan 21, 2011 at 12:28 AM, Tom Lane wrote: Magnus Hagander writes: >> - Why not initialize logid and logseg like

Re: [HACKERS] Include WAL in base backup

2011-01-24 Thread Fujii Masao
On Mon, Jan 24, 2011 at 4:47 PM, Magnus Hagander wrote: > On Mon, Jan 24, 2011 at 08:45, Fujii Masao wrote: >> On Fri, Jan 21, 2011 at 12:28 AM, Tom Lane wrote: >>> Magnus Hagander writes: > - Why not initialize logid and logseg like so?: > >        int logid = startptr.xlogid;

Re: [HACKERS] Include WAL in base backup

2011-01-23 Thread Magnus Hagander
On Mon, Jan 24, 2011 at 08:45, Fujii Masao wrote: > On Fri, Jan 21, 2011 at 12:28 AM, Tom Lane wrote: >> Magnus Hagander writes: - Why not initialize logid and logseg like so?:        int logid = startptr.xlogid;        int logseg = startptr.xrecoff / XLogSegSize;  

Re: [HACKERS] Include WAL in base backup

2011-01-23 Thread Fujii Masao
On Fri, Jan 21, 2011 at 12:28 AM, Tom Lane wrote: > Magnus Hagander writes: >>> - Why not initialize logid and logseg like so?: >>> >>>        int logid = startptr.xlogid; >>>        int logseg = startptr.xrecoff / XLogSegSize; >>> >>>  Then use those in your elog?  Seems cleaner to me. > >> Hmm.

Re: [HACKERS] Include WAL in base backup

2011-01-20 Thread Tom Lane
Magnus Hagander writes: >> - Why not initialize logid and logseg like so?: >> >>        int logid = startptr.xlogid; >>        int logseg = startptr.xrecoff / XLogSegSize; >> >>  Then use those in your elog?  Seems cleaner to me. > Hmm. Yes. Agreed. Marginal complaint here: int is the wrong ty

Re: [HACKERS] Include WAL in base backup

2011-01-20 Thread Magnus Hagander
On Thu, Jan 20, 2011 at 05:03, Stephen Frost wrote: > Greetings, > > * Magnus Hagander (mag...@hagander.net) wrote: >> For now, you need to set wal_keep_segments to make it work properly, >> but if you do the idea is that the tar file/stream generated in the >> base backup will include all the req

Re: [HACKERS] Include WAL in base backup

2011-01-19 Thread Stephen Frost
Magnus, * Stephen Frost (sfr...@snowman.net) wrote: > mkay, I'm not going to try to make this ready for committer, but will > provide my comments on it overall. Bit difficult to review when someone > else is reviewing the base patch too. :/ I went ahead and marked it as 'waiting on author', sinc

Re: [HACKERS] Include WAL in base backup

2011-01-19 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > For now, you need to set wal_keep_segments to make it work properly, > but if you do the idea is that the tar file/stream generated in the > base backup will include all the required WAL files. Is there some reason to not ERROR outright

Re: [HACKERS] Include WAL in base backup

2011-01-17 Thread Dimitri Fontaine
Magnus Hagander writes: > But however we do it, it will be significantly more complex than just > including the WAL. And I want to make sure we get *something* done in > time for 9.1, and then improve upon it. If we can get the improvement > into 9.1 that's great, but if not it will have to wait u

Re: [HACKERS] Include WAL in base backup

2011-01-17 Thread Magnus Hagander
On Sun, Jan 16, 2011 at 20:13, Dimitri Fontaine wrote: > Magnus Hagander writes: >> However, it's not quite that simple. "just adding a fork()" doesn't >> work on all our platforms, and you need to deal with feedback and such >> between them as well. > > I'd think client-side, we have an existing

Re: [HACKERS] Include WAL in base backup

2011-01-16 Thread Dimitri Fontaine
Magnus Hagander writes: > However, it's not quite that simple. "just adding a fork()" doesn't > work on all our platforms, and you need to deal with feedback and such > between them as well. I'd think client-side, we have an existing implementation with the parallel pg_restore option. Don't know

Re: [HACKERS] Include WAL in base backup

2011-01-16 Thread Magnus Hagander
On Sun, Jan 16, 2011 at 18:45, Dimitri Fontaine wrote: > Magnus Hagander writes: >>> What if you start a concurrent process that begins streaming the WAL >>> segments just before you start the backup, and you stop it after having >>> stopped the backup.  I would think that then, the local receive

Re: [HACKERS] Include WAL in base backup

2011-01-16 Thread Dimitri Fontaine
Magnus Hagander writes: >> What if you start a concurrent process that begins streaming the WAL >> segments just before you start the backup, and you stop it after having >> stopped the backup.  I would think that then, the local received files >> would be complete.  We would only need a program a

Re: [HACKERS] Include WAL in base backup

2011-01-15 Thread Magnus Hagander
On Sat, Jan 15, 2011 at 23:32, Dimitri Fontaine wrote: > Magnus Hagander writes: >> Here's a cutdown version of the idea about including WAL in the base >> backup. What I initially wanted was to introduce a way to guarantee >> that the required WAL (with some sort of cutoff of course) would be >>

Re: [HACKERS] Include WAL in base backup

2011-01-15 Thread Dimitri Fontaine
Magnus Hagander writes: > Here's a cutdown version of the idea about including WAL in the base > backup. What I initially wanted was to introduce a way to guarantee > that the required WAL (with some sort of cutoff of course) would be > available for the backup, but I ran out of time for that. We

[HACKERS] Include WAL in base backup

2011-01-15 Thread Magnus Hagander
Here's a cutdown version of the idea about including WAL in the base backup. What I initially wanted was to introduce a way to guarantee that the required WAL (with some sort of cutoff of course) would be available for the backup, but I ran out of time for that. We can always add that later. For n