Re: [HACKERS] Online base backup from the hot-standby

2012-01-25 Thread Fujii Masao
On Thu, Jan 26, 2012 at 3:07 AM, Simon Riggs wrote: > On Wed, Jan 25, 2012 at 8:49 AM, Simon Riggs wrote: >> On Wed, Jan 25, 2012 at 8:16 AM, Fujii Masao wrote: >> What happens if we shutdown the WALwriter and then issue SIGHUP? >>> >>> SIGHUP doesn't affect full_page_writes in that case. O

Re: [HACKERS] Online base backup from the hot-standby

2012-01-25 Thread Simon Riggs
On Wed, Jan 25, 2012 at 8:49 AM, Simon Riggs wrote: > On Wed, Jan 25, 2012 at 8:16 AM, Fujii Masao wrote: > >>> What happens if we shutdown the WALwriter and then issue SIGHUP? >> >> SIGHUP doesn't affect full_page_writes in that case. Oh, you are concerned >> about >> the case where smart shutd

Re: [HACKERS] Online base backup from the hot-standby

2012-01-25 Thread Simon Riggs
On Wed, Jan 25, 2012 at 8:16 AM, Fujii Masao wrote: >> What happens if we shutdown the WALwriter and then issue SIGHUP? > > SIGHUP doesn't affect full_page_writes in that case. Oh, you are concerned > about > the case where smart shutdown kills walwriter but some backends are > still running? >

Re: [HACKERS] Online base backup from the hot-standby

2012-01-25 Thread Fujii Masao
On Tue, Jan 24, 2012 at 7:54 PM, Simon Riggs wrote: > On Tue, Jan 24, 2012 at 9:51 AM, Fujii Masao wrote: > >>> I'll proceed to commit for this now. >> >> Thanks a lot! > > Can I just check a few things? Sure! > You say > /* > +        * Update full_page_writes in shared memory and write an > +

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 10:54 AM, Simon Riggs wrote: > On Tue, Jan 24, 2012 at 9:51 AM, Fujii Masao wrote: > >>> I'll proceed to commit for this now. >> >> Thanks a lot! > > Can I just check a few things? Just to clarify, not expecting another patch version, just reply here and I can edit. --

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 9:51 AM, Fujii Masao wrote: >> I'll proceed to commit for this now. > > Thanks a lot! Can I just check a few things? You say /* +* Update full_page_writes in shared memory and write an +* XLOG_FPW_CHANGE record before resource manager writes cleanup +

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Fujii Masao
On Mon, Jan 23, 2012 at 10:11 PM, Simon Riggs wrote: > On Mon, Jan 23, 2012 at 10:29 AM, Fujii Masao wrote: >> On Fri, Jan 20, 2012 at 11:34 PM, Simon Riggs wrote: >>> On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao wrote: Thanks for the review! On Fri, Jan 20, 2012 at 8:15 PM, Sim

Re: [HACKERS] Online base backup from the hot-standby

2012-01-23 Thread Robert Haas
On Mon, Jan 23, 2012 at 8:11 AM, Robert Haas wrote: > On Mon, Jan 23, 2012 at 5:29 AM, Fujii Masao wrote: >> If many people think the patch is not acceptable without such a safeguard, >> I will do that right now. > > That's my view.  I think we ought to resolve this issue before commit, > especia

Re: [HACKERS] Online base backup from the hot-standby

2012-01-23 Thread Robert Haas
On Mon, Jan 23, 2012 at 5:29 AM, Fujii Masao wrote: > If many people think the patch is not acceptable without such a safeguard, > I will do that right now. That's my view. I think we ought to resolve this issue before commit, especially since it seems unclear that we know how to fix it. -- Ro

Re: [HACKERS] Online base backup from the hot-standby

2012-01-23 Thread Simon Riggs
On Mon, Jan 23, 2012 at 10:29 AM, Fujii Masao wrote: > On Fri, Jan 20, 2012 at 11:34 PM, Simon Riggs wrote: >> On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao wrote: >>> Thanks for the review! >>> >>> On Fri, Jan 20, 2012 at 8:15 PM, Simon Riggs wrote: I'm looking at this patch and wondering

Re: [HACKERS] Online base backup from the hot-standby

2012-01-23 Thread Fujii Masao
On Fri, Jan 20, 2012 at 11:34 PM, Simon Riggs wrote: > On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao wrote: >> Thanks for the review! >> >> On Fri, Jan 20, 2012 at 8:15 PM, Simon Riggs wrote: >>> I'm looking at this patch and wondering why we're doing so many >>> press-ups to ensure full_page_wr

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Simon Riggs
On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao wrote: > Thanks for the review! > > On Fri, Jan 20, 2012 at 8:15 PM, Simon Riggs wrote: >> I'm looking at this patch and wondering why we're doing so many >> press-ups to ensure full_page_writes parameter is on. This will still >> fail if you use a ut

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Fujii Masao
Thanks for the review! On Fri, Jan 20, 2012 at 8:15 PM, Simon Riggs wrote: > I'm looking at this patch and wondering why we're doing so many > press-ups to ensure full_page_writes parameter is on. This will still > fail if you use a utility that removes the full page writes, but fail > silently.

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Simon Riggs
On Fri, Jan 20, 2012 at 11:04 AM, Fujii Masao wrote: > But Steve encountered it again, which means that the above fix is not > sufficient. Unless the issue is derived from my patch, we should do > another cycle of diagnosis of it. It's my bug, and I've posted a fix but not yet applied it, just a

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Simon Riggs
On Tue, Jan 17, 2012 at 10:38 AM, Fujii Masao wrote: > On Fri, Jan 13, 2012 at 5:02 PM, Fujii Masao wrote: >> The amount of code changes to allow pg_basebackup to make a backup from >> the standby seems to be small. So I ended up merging that changes and the >> infrastructure patch. WIP patch att

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Fujii Masao
On Fri, Jan 20, 2012 at 7:37 PM, Erik Rijkers wrote: > I'm not sure, but it does look like this is the "mystery" bug that I > encountered repeatedly > already in 9.0devel; but I was never able to reproduce it reliably.  But I > don't think it was ever > solved. > >  http://archives.postgresql.or

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Erik Rijkers
On Fri, January 20, 2012 05:01, Steve Singer wrote: > On 12-01-17 05:38 AM, Fujii Masao wrote: >> On Fri, Jan 13, 2012 at 5:02 PM, Fujii Masao wrote: >>> The amount of code changes to allow pg_basebackup to make a backup from >>> the standby seems to be small. So I ended up merging that changes an

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Fujii Masao
On Fri, Jan 20, 2012 at 1:01 PM, Steve Singer wrote: > Here is my review of this verison of the patch. I think this patch has been > in every CF for 9.2 and I feel it is getting close to being committed. Thanks for the review! > Testing Review > > > I encountered

Re: [HACKERS] Online base backup from the hot-standby

2012-01-19 Thread Steve Singer
On 12-01-17 05:38 AM, Fujii Masao wrote: On Fri, Jan 13, 2012 at 5:02 PM, Fujii Masao wrote: The amount of code changes to allow pg_basebackup to make a backup from the standby seems to be small. So I ended up merging that changes and the infrastructure patch. WIP patch attached. But I'd happy

Re: [HACKERS] Online base backup from the hot-standby

2011-11-14 Thread Steve Singer
On 11-10-31 12:11 AM, Jun Ishiduka wrote: >> >> Agreed. I'll extract FPW stuff from the patch that I submitted, and revise it >> as the infrastructure patch. >> >> The changes of pg_start_backup() etc that Ishiduka-san did are also >> a server-side infrastructure. I will extract them as another inf

Re: [HACKERS] Online base backup from the hot-standby

2011-11-03 Thread Fujii Masao
On Fri, Nov 4, 2011 at 8:06 AM, Josh Berkus wrote: > On 10/25/11 5:03 AM, Magnus Hagander wrote: >> If we want something to go in early, that could be as simple as a >> version of pg_basebackup that runs against the slave but only if >> full_page_writes=on on the master. If it's not, it throws an

Re: [HACKERS] Online base backup from the hot-standby

2011-11-03 Thread Josh Berkus
On 10/25/11 5:03 AM, Magnus Hagander wrote: > If we want something to go in early, that could be as simple as a > version of pg_basebackup that runs against the slave but only if > full_page_writes=on on the master. If it's not, it throws an error. > Then we can improve upon that by adding handling

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Fujii Masao
On Tue, Oct 25, 2011 at 9:03 PM, Magnus Hagander wrote: > On Tue, Oct 25, 2011 at 13:54, Fujii Masao wrote: >> On Tue, Oct 25, 2011 at 7:19 PM, Magnus Hagander wrote: >>> I don't think we should necessarily give up completely. But doing a >>> pg_basebackup way *first* seems reasonable - because

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Heikki Linnakangas
On 25.10.2011 15:56, Steve Singer wrote: On 11-10-25 02:44 AM, Heikki Linnakangas wrote: With pg_basebackup, we have a fighting chance of getting this right, because we have more control over how the backup is made. For example, we can co-operate with the buffer manager to avoid torn-pages, elim

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Steve Singer
On 11-10-25 02:44 AM, Heikki Linnakangas wrote: With pg_basebackup, we have a fighting chance of getting this right, because we have more control over how the backup is made. For example, we can co-operate with the buffer manager to avoid torn-pages, eliminating the need for full_page_writes=on

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Magnus Hagander
On Tue, Oct 25, 2011 at 13:54, Fujii Masao wrote: > On Tue, Oct 25, 2011 at 7:19 PM, Magnus Hagander wrote: >> I don't think we should necessarily give up completely. But doing a >> pg_basebackup way *first* seems reasonable - because it's going to be >> the easiest one to "get right", given that

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Fujii Masao
On Tue, Oct 25, 2011 at 7:19 PM, Magnus Hagander wrote: > I don't think we should necessarily give up completely. But doing a > pg_basebackup way *first* seems reasonable - because it's going to be > the easiest one to "get right", given that we have more control there. > Doesn't mean we shouldn't

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Magnus Hagander
On Tue, Oct 25, 2011 at 10:50, Fujii Masao wrote: > On Tue, Oct 25, 2011 at 3:44 PM, Heikki Linnakangas > wrote: > + > +      Again connect to the database as a superuser, and execute > +pg_stop_backup. This terminates the backup mode, but > does not > +      perform a switch

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Fujii Masao
On Tue, Oct 25, 2011 at 3:44 PM, Heikki Linnakangas wrote: + +      Again connect to the database as a superuser, and execute +pg_stop_backup. This terminates the backup mode, but does not +      perform a switch to the next WAL segment, create a backup history file a

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Heikki Linnakangas
On 25.10.2011 08:12, Fujii Masao wrote: On Tue, Oct 25, 2011 at 12:24 AM, Heikki Linnakangas wrote: On 24.10.2011 15:29, Fujii Masao wrote: + + + Copy the pg_control file from the cluster directory to the global + sub-directory of the backup. For example: + + cp $PGDATA/global/pg_c

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Fujii Masao
On Tue, Oct 25, 2011 at 12:33 AM, Heikki Linnakangas wrote: > One problem with this whole FPW-tracking is that pg_lesslog makes it fail. > I'm not sure what we need to do about that - maybe just add a warning to the > docs. But it leaves a bit bad feeling in my mouth. Usually we try to make > feat

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Fujii Masao
Thanks for the review! On Tue, Oct 25, 2011 at 12:24 AM, Heikki Linnakangas wrote: > On 24.10.2011 15:29, Fujii Masao wrote: >> >> +     >> +     >> +      Copy the pg_control file from the cluster directory to the global >> +      sub-directory of the backup. For example: >> + >> + cp $PGDATA/

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Robert Haas
On Mon, Oct 24, 2011 at 11:33 AM, Heikki Linnakangas wrote: > On 24.10.2011 15:29, Fujii Masao wrote: >> >> In your patch, FPW is always WAL-logged at startup even when FPW has >> not been changed since last shutdown. I don't think that's required. >> I changed the recovery code so that it keeps t

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Heikki Linnakangas
On 24.10.2011 15:29, Fujii Masao wrote: In your patch, FPW is always WAL-logged at startup even when FPW has not been changed since last shutdown. I don't think that's required. I changed the recovery code so that it keeps track of last FPW indicated by WAL record. Then, at end of startup, if tha

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Heikki Linnakangas
On 24.10.2011 15:29, Fujii Masao wrote: + + + Copy the pg_control file from the cluster directory to the global + sub-directory of the backup. For example: + + cp $PGDATA/global/pg_control /mnt/server/backupdir/global + + + Why is this step required? The control fi

Re: [HACKERS] Online base backup from the hot-standby

2011-10-19 Thread Jun Ishiduka
> As I suggested in the reply to Simon, I think that the change of FPW > should be WAL-logged separately from that of HS parameters. ISTM > packing them in one WAL record makes XLogReportParameters() > quite confusing. Thought? I updated a patch for what you have suggested (that the change of FPW

Re: [HACKERS] Online base backup from the hot-standby

2011-10-18 Thread Jun Ishiduka
> > + /* > > +* The backend writes WAL of FPW at checkpoint. However, The > > backend do > > +* not need to write WAL of FPW at checkpoint shutdown because > > it > > +* performs when startup finishes. > > +*/ > > + XLogReportPa

Re: [HACKERS] Online base backup from the hot-standby

2011-10-17 Thread Jun Ishiduka
> + /* > + * The backend writes WAL of FPW at checkpoint. However, The > backend do > + * not need to write WAL of FPW at checkpoint shutdown because > it > + * performs when startup finishes. > + */ > + XLogReportParame

Re: [HACKERS] Online base backup from the hot-standby

2011-10-17 Thread Fujii Masao
2011/10/15 Jun Ishiduka : > >> >     if (!shutdown && XLogStandbyInfoActive()) >> > +   { >> >             LogStandbySnapshot(&checkPoint.oldestActiveXid, >> > &checkPoint.nextXid); >> > +           XLogReportParameters(REPORT_ON_BACKEND); >> > +   } >> > >> > Why doesn't the change of FPW need to

Re: [HACKERS] Online base backup from the hot-standby

2011-10-14 Thread Jun Ishiduka
> > if (!shutdown && XLogStandbyInfoActive()) > > + { > > LogStandbySnapshot(&checkPoint.oldestActiveXid, > > &checkPoint.nextXid); > > + XLogReportParameters(REPORT_ON_BACKEND); > > + } > > > > Why doesn't the change of FPW need to be WAL-logged when > > shutdown c

Re: [HACKERS] Online base backup from the hot-standby

2011-10-14 Thread Jun Ishiduka
> As I suggested in the reply to Simon, I think that the change of FPW > should be WAL-logged separately from that of HS parameters. ISTM > packing them in one WAL record makes XLogReportParameters() > quite confusing. Thought? I want to confirm the reply of Simon. I think we cannot decide how th

Re: [HACKERS] Online base backup from the hot-standby

2011-10-14 Thread Fujii Masao
2011/10/13 Jun Ishiduka : > I updated to patch corresponded above-comments. Thanks for updating the patch! As I suggested in the reply to Simon, I think that the change of FPW should be WAL-logged separately from that of HS parameters. ISTM packing them in one WAL record makes XLogReportParameter

Re: [HACKERS] Online base backup from the hot-standby

2011-10-13 Thread Fujii Masao
On Mon, Oct 10, 2011 at 3:56 AM, Simon Riggs wrote: > 2011/10/9 Jun Ishiduka : > >>  Insert WAL including a value of current FPW (on master) >>   * In the the same timing as update, they insert WAL (is named >>     XLOG_FPW_CHANGE). XLOG_FPW_CHANGE has a value of the changed FPW. >>   * When it cr

Re: [HACKERS] Online base backup from the hot-standby

2011-10-13 Thread Jun Ishiduka
> > > > > ERROR: full_page_writes on master is set invalid at least once since > > > > latest checkpoint > > > > > > > > I think this error should be rewritten as > > > > ERROR: full_page_writes on master has been off at some point since > > > > latest checkpoint > > > > > > > > We should be usin

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
Sorry. I was not previously able to answer fujii's all comments. This is the remaining answers. > + LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); > + XLogCtl->Insert.fullPageWrites = fullPageWrites; > + LWLockRelease(WALInsertLock); > > I don't think WALInsertLock needs to be hold her

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
> > > ERROR: full_page_writes on master is set invalid at least once since > > > latest checkpoint > > > > > > I think this error should be rewritten as > > > ERROR: full_page_writes on master has been off at some point since > > > latest checkpoint > > > > > > We should be using 'off' instead of

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Fujii Masao
2011/10/12 Jun Ishiduka : > > ERROR: full_page_writes on master is set invalid at least once since > > latest checkpoint > > > > I think this error should be rewritten as > > ERROR: full_page_writes on master has been off at some point since > > latest checkpoint > > > > We should be using 'off' in

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
> > Some testing notes > > -- > > select pg_start_backup('x'); > > ERROR: full_page_writes on master is set invalid at least once since > > latest checkpoint > > > > I think this error should be rewritten as > > ERROR: full_page_writes on master has been off at some po

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
> Some testing notes > -- > select pg_start_backup('x'); > ERROR: full_page_writes on master is set invalid at least once since > latest checkpoint > > I think this error should be rewritten as > ERROR: full_page_writes on master has been off at some point since > late

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Steve Singer
On 11-10-11 11:17 AM, Jun Ishiduka wrote: > Done. > > Updated patch attached. > I have taken Jun's latest patch and applied it on top of Fujii's most recent patch. I did some testing with the result but nothing theory enough to stumble on any race conditions. Some testing notes --

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
> > I can't see a reason why we would use a new WAL record for this, > > rather than modify the XLOG_PARAMETER_CHANGE record type which was > > created for a very similar reason. > > The code would be much simpler if we just extend > > XLOG_PARAMETER_CHANGE, so please can we do that? > > Sure. >

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
> I can't see a reason why we would use a new WAL record for this, > rather than modify the XLOG_PARAMETER_CHANGE record type which was > created for a very similar reason. > The code would be much simpler if we just extend > XLOG_PARAMETER_CHANGE, so please can we do that? Sure. > The log messa

Re: [HACKERS] Online base backup from the hot-standby

2011-10-09 Thread Simon Riggs
2011/10/9 Jun Ishiduka : >  Insert WAL including a value of current FPW (on master) >   * In the the same timing as update, they insert WAL (is named >     XLOG_FPW_CHANGE). XLOG_FPW_CHANGE has a value of the changed FPW. >   * When it creates CHECKPOINT, it adds a value of current FPW to the >  

Re: [HACKERS] Online base backup from the hot-standby

2011-10-09 Thread Jun Ishiduka
I created a patch corresponding FPW. Fujii's patch (ver 9) is based. Manage own FPW in shared-memory (on master) * startup and walwriter process update it. startup initializes it after REDO. walwriter updates it when started or received SIGHUP. Insert WAL including a value of current F

Re: [HACKERS] Online base backup from the hot-standby

2011-09-27 Thread Fujii Masao
On Wed, Sep 28, 2011 at 8:10 AM, Steve Singer wrote: > This is the test procedure I'm trying today, I wasn't able to reproduce the > crash.  What I was doing the other day was similar but I can't speak to > unintentional differences. Thanks for the info! I tried your test case three times, but wa

Re: [HACKERS] Online base backup from the hot-standby

2011-09-27 Thread Steve Singer
On 11-09-26 10:56 PM, Fujii Masao wrote: Looks weired. Though the WAL record starting from 0/6000298 was read successfully, then re-fetch of the same record fails at the end of recovery. One possible cause is the corruption of archived WAL file. What restore_command on the standby and archive_co

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Fujii Masao
On Tue, Sep 27, 2011 at 11:56 AM, Fujii Masao wrote: >> In backup.sgml  the new section titled "Making a Base Backup during >> Recovery"  I would prefer to see some mention in the title that this >> procedure is for standby servers ie "Making a Base Backup from a Standby >> Database".  Users who h

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Fujii Masao
On Mon, Sep 26, 2011 at 11:39 AM, Steve Singer wrote: > I have looked at both Jun's patch from Sept 13 and Fujii's updates to the > patch.  I agree that Fujii's updated version should be used as the basis for > changes going forward.   My comments below refer to that version (unless > otherwise no

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Fujii Masao
On Fri, Sep 23, 2011 at 12:44 AM, Magnus Hagander wrote: > Would it make sense for pg_start_backup() to have the ability to wait > for the next restartpoint in a case like this, if we know that FPW has > been set? Instead of failing? Or maybe that's just overcomplicating > things when trying to be

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Jun Ishiduka
> Attached is the updated version of the patch. I refactored the code, fixed > some bugs, added lots of source code comments, improved the document, > but didn't change the basic design. Please check this patch, and let's use > this patch as the base if you agree with that. Thanks for update patch

Re: [HACKERS] Online base backup from the hot-standby

2011-09-25 Thread Steve Singer
On 11-09-22 09:24 AM, Fujii Masao wrote: On Wed, Sep 21, 2011 at 11:50 AM, Fujii Masao wrote: 2011/9/13 Jun Ishiduka: Update patch. Changes: * set 'on' full_page_writes by user (in document) * read "FROM: XX" in backup_label (in xlog.c) * check status when pg_stop_backup is executed (in

Re: [HACKERS] Online base backup from the hot-standby

2011-09-22 Thread Magnus Hagander
On Thu, Sep 22, 2011 at 14:13, Fujii Masao wrote: > On Wed, Sep 21, 2011 at 5:34 PM, Magnus Hagander wrote: >> On Wed, Sep 21, 2011 at 08:23, Fujii Masao wrote: >>> On Wed, Sep 21, 2011 at 2:13 PM, Magnus Hagander >>> wrote: Presumably pg_start_backup() will check this. And we'll somehow

Re: [HACKERS] Online base backup from the hot-standby

2011-09-22 Thread Fujii Masao
On Wed, Sep 21, 2011 at 11:50 AM, Fujii Masao wrote: > 2011/9/13 Jun Ishiduka : >> >> Update patch. >> >> Changes: >>  * set 'on' full_page_writes by user (in document) >>  * read "FROM: XX" in backup_label (in xlog.c) >>  * check status when pg_stop_backup is executed (in xlog.c) > > Thanks for u

Re: [HACKERS] Online base backup from the hot-standby

2011-09-22 Thread Fujii Masao
On Wed, Sep 21, 2011 at 5:34 PM, Magnus Hagander wrote: > On Wed, Sep 21, 2011 at 08:23, Fujii Masao wrote: >> On Wed, Sep 21, 2011 at 2:13 PM, Magnus Hagander wrote: >>> Presumably pg_start_backup() will check this. And we'll somehow track >>> this before pg_stop_backup() as well? (for such evi

Re: [HACKERS] Online base backup from the hot-standby

2011-09-21 Thread Magnus Hagander
On Wed, Sep 21, 2011 at 08:23, Fujii Masao wrote: > On Wed, Sep 21, 2011 at 2:13 PM, Magnus Hagander wrote: >> On Wed, Sep 21, 2011 at 04:50, Fujii Masao wrote: >>> 3. Copy the pg_control file from the cluster directory on the standby to >>>    the backup as follows: >>> >>>    cp $PGDATA/global

Re: [HACKERS] Online base backup from the hot-standby

2011-09-20 Thread Fujii Masao
On Wed, Sep 21, 2011 at 2:13 PM, Magnus Hagander wrote: > On Wed, Sep 21, 2011 at 04:50, Fujii Masao wrote: >> 3. Copy the pg_control file from the cluster directory on the standby to >>    the backup as follows: >> >>    cp $PGDATA/global/pg_control /mnt/server/backupdir/global > > But this is d

Re: [HACKERS] Online base backup from the hot-standby

2011-09-20 Thread Magnus Hagander
On Wed, Sep 21, 2011 at 04:50, Fujii Masao wrote: > 2011/9/13 Jun Ishiduka : >> >> Update patch. >> >> Changes: >>  * set 'on' full_page_writes by user (in document) >>  * read "FROM: XX" in backup_label (in xlog.c) >>  * check status when pg_stop_backup is executed (in xlog.c) > > Thanks for upda

Re: [HACKERS] Online base backup from the hot-standby

2011-09-20 Thread Fujii Masao
2011/9/13 Jun Ishiduka : > > Update patch. > > Changes: >  * set 'on' full_page_writes by user (in document) >  * read "FROM: XX" in backup_label (in xlog.c) >  * check status when pg_stop_backup is executed (in xlog.c) Thanks for updating the patch. Before reviewing the patch, to encourage peopl

Re: [HACKERS] Online base backup from the hot-standby

2011-09-13 Thread Jun Ishiduka
Update patch. Changes: * set 'on' full_page_writes by user (in document) * read "FROM: XX" in backup_label (in xlog.c) * check status when pg_stop_backup is executed (in xlog.c) > Hi, Created a patch in response to comments. > > > * Procedure > 1. Call pg_start_backup('x') on hot standby

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Jun Ishiduka
> > * Procedure > > > > 1. Call pg_start_backup('x') on the standby. > > 2. Take a backup of the data dir. > > 3. Call pg_stop_backup() on the standby. > > 4. Copy the control file on the standby to the backup. > > 5. Check whether the control file is status during hot standby with > > pg_control

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Fujii Masao
2011/8/5 Jun Ishiduka : > * Procedure > > 1. Call pg_start_backup('x') on the standby. > 2. Take a backup of the data dir. > 3. Call pg_stop_backup() on the standby. > 4. Copy the control file on the standby to the backup. > 5. Check whether the control file is status during hot standby with > pg_

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Fujii Masao
On Thu, Aug 18, 2011 at 12:09 AM, Robert Haas wrote: > Ugh, you're right.  But then you might have problems if the state > changes again before all backends have picked up the previous change. Right. > What I've thought about before is making one backend (say, bgwriter) > store its latest value

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Robert Haas
On Wed, Aug 17, 2011 at 9:53 AM, Fujii Masao wrote: > On Wed, Aug 17, 2011 at 9:40 PM, Robert Haas wrote: >> On Wed, Aug 17, 2011 at 6:19 AM, Fujii Masao wrote: >>> The straightforward approach to address the problem you raised is to log >>> the change of full_page_writes on the master. Since su

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Fujii Masao
On Wed, Aug 17, 2011 at 9:40 PM, Robert Haas wrote: > On Wed, Aug 17, 2011 at 6:19 AM, Fujii Masao wrote: >> The straightforward approach to address the problem you raised is to log >> the change of full_page_writes on the master. Since such a WAL record is also >> replicated to the standby, the

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Robert Haas
On Wed, Aug 17, 2011 at 6:19 AM, Fujii Masao wrote: > 2011/8/17 Jun Ishiduka : >>> I see in xlog.h XLR_BKP_REMOVABLE, the comment above it says that this >>> flag is used to indicate that the archiver can compress the full page >>> blocks to non-full page blocks. I am not familiar with where in th

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Fujii Masao
2011/8/17 Jun Ishiduka : >> I see in xlog.h XLR_BKP_REMOVABLE, the comment above it says that this >> flag is used to indicate that the archiver can compress the full page >> blocks to non-full page blocks. I am not familiar with where in the code >> this actually happens but will this cause issues

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Jun Ishiduka
> Is there any way to tell from the WAL segments if they contain the full > page data? If so could you verify this on the second slave when it is > brought up? Or can you track this on the first slave and produce an > error in either pg_start_backup or pg_stop_backup() Sure. I will make a patch w

Re: [HACKERS] Online base backup from the hot-standby

2011-08-16 Thread Steve Singer
On 11-08-16 02:09 AM, Jun Ishiduka wrote: > > Thanks. > > This has the following two problems. > * pg_start_backup() must set 'on' to full_page_writes of the master that >is actual writing of the WAL, but not the standby. Is there any way to tell from the WAL segments if they contain the ful

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Jun Ishiduka
> >> > * Not correspond yet > >> > > >> > ?* full_page_write = off > >> > ? ?-> If the primary is "full_page_write = off", archive recovery may > >> > not act > >> > ? ? ? normally. Therefore the standby may need to check whether > >> > "full_page_write > >> > ? ? ? = off" to WAL. > >> > >> Isn'

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Jun Ishiduka
> >> > * Not correspond yet > >> > > >> > ?* full_page_write = off > >> > ? ?-> If the primary is "full_page_write = off", archive recovery may > >> > not act > >> > ? ? ? normally. Therefore the standby may need to check whether > >> > "full_page_write > >> > ? ? ? = off" to WAL. > >> > >> Isn'

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Robert Haas
2011/8/15 Jun Ishiduka : >> > * Not correspond yet >> > >> >  * full_page_write = off >> >    -> If the primary is "full_page_write = off", archive recovery may not >> > act >> >       normally. Therefore the standby may need to check whether >> > "full_page_write >> >       = off" to WAL. >> >>

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Jun Ishiduka
> > * Not correspond yet > > > > * full_page_write = off > >-> If the primary is "full_page_write = off", archive recovery may not > > act > > normally. Therefore the standby may need to check whether > > "full_page_write > > = off" to WAL. > > Isn't having a standby make the f

Re: [HACKERS] Online base backup from the hot-standby

2011-08-05 Thread Cédric Villemain
2011/8/5 Jun Ishiduka : >> I will provide a patch which can exeute pg_start/stop_backup >> including to solve above comment and conditions in next stage. >> Then please review. > > done. great ! > > > * Procedure > > 1. Call pg_start_backup('x') on the standby. > 2. Take a backup of the data dir.

[HACKERS] Online base backup from the hot-standby

2011-08-04 Thread Jun Ishiduka
> I will provide a patch which can exeute pg_start/stop_backup > including to solve above comment and conditions in next stage. > Then please review. done. * Procedure 1. Call pg_start_backup('x') on the standby. 2. Take a backup of the data dir. 3. Call pg_stop_backup() on the standby. 4. Copy

Re: [HACKERS] Online base backup from the hot-standby

2011-07-12 Thread Jun Ishiduka
> This version of the patch adds a field into pg_controldata that tries to > store the source of the base backup while in recovery mode. > I think your ultimate goal with this patch is to be able to take a > backup of a running hot-standby slave and recover it as another > instance. This patch see

Re: [HACKERS] Online base backup from the hot-standby

2011-07-10 Thread Steve Singer
On 11-07-07 09:22 PM, Jun Ishiduka wrote: >> As you proposed, adding new field which stores the backup end location >> taken from minRecoveryPoint, into pg_control sounds good idea. > Update patch. > Here is a review of the updated patch This version of the patch adds a field into pg_controldata t

Re: [HACKERS] Online base backup from the hot-standby

2011-07-07 Thread Jun Ishiduka
> As you proposed, adding new field which stores the backup end location > taken from minRecoveryPoint, into pg_control sounds good idea. Update patch. Regards. Jun Ishizuka NTT Software Corporation TEL:045-317-7018 E-Mail: ishizuka@po.ntts.co.jp

Re: [HACKERS] Online base backup from the hot-standby

2011-07-05 Thread Fujii Masao
2011/7/5 Jun Ishiduka : > >> What about using backupStartPoint to check whether this recovery >> started from the backup or not? > > No, postgres can check whether this recovery started from the backup > or not, but can not check whether standby server or master (got backup > from). Oh, right. We

Re: [HACKERS] Online base backup from the hot-standby

2011-07-05 Thread Jun Ishiduka
> What about using backupStartPoint to check whether this recovery > started from the backup or not? No, postgres can check whether this recovery started from the backup or not, but can not check whether standby server or master (got backup from). Once recovery started, backupStartPoint is rec

Re: [HACKERS] Online base backup from the hot-standby

2011-07-04 Thread Fujii Masao
2011/7/4 Jun Ishiduka : > >> When the standby restarts after it crashes during recovery, it always >> checks whether recovery has reached the backup end location by >> using minRecoveryPoint even though the standby doesn't start from >> the backup. This looks odd. > > Certainly. > > But, in this ca

Re: [HACKERS] Online base backup from the hot-standby

2011-07-04 Thread Jun Ishiduka
> When the standby restarts after it crashes during recovery, it always > checks whether recovery has reached the backup end location by > using minRecoveryPoint even though the standby doesn't start from > the backup. This looks odd. Certainly. But, in this case, the state before recovery start

Re: [HACKERS] Online base backup from the hot-standby

2011-07-03 Thread Fujii Masao
2011/7/1 Jun Ishiduka : > >> On this commitfest, the goal of the patch is to be able to be >> recovered using "Minimum recovery ending location" in the control file. > > Done. When the standby restarts after it crashes during recovery, it always checks whether recovery has reached the backup end l

Re: [HACKERS] Online base backup from the hot-standby

2011-06-30 Thread Jun Ishiduka
> On this commitfest, the goal of the patch is to be able to be > recovered using "Minimum recovery ending location" in the control file. Done. Regards. Jun Ishizuka NTT Software Corporation TEL:045-317-7018 E-Mail: ishizuka@po.ntts.co.jp --

Re: [HACKERS] Online base backup from the hot-standby

2011-06-29 Thread Jun Ishiduka
> > > Process of online base backup on standby server: > > > 1. pg_start_backup('x'); > > > 2. copy the data directory > > > 3. copy *pg_control* > > > > Who deletes the backup_label file created by pg_start_backup()? > > Isn't pg_stop_backup() required to do that? > > You need it to take the

Re: [HACKERS] Online base backup from the hot-standby

2011-06-29 Thread Magnus Hagander
On Jun 30, 2011 5:59 AM, "Fujii Masao" wrote: > > 2011/6/28 Jun Ishiduka : > > > >> Considering everything that has been discussed on this thread so far. > >> > >> Do you still think your patch is the best way to accomplish base backups > >> from standby servers? > >> If not what changes do you th

Re: [HACKERS] Online base backup from the hot-standby

2011-06-29 Thread Fujii Masao
2011/6/28 Jun Ishiduka : > >> Considering everything that has been discussed on this thread so far. >> >> Do you still think your patch is the best way to accomplish base backups >> from standby servers? >> If not what changes do you think should be made? > > I reconsider the way to not use pg_stop

Re: [HACKERS] Online base backup from the hot-standby

2011-06-28 Thread Steve Singer
On 11-06-28 01:52 AM, Jun Ishiduka wrote: >> Considering everything that has been discussed on this thread so far. >> >> Do you still think your patch is the best way to accomplish base backups >> from standby servers? >> If not what changes do you think should be made? > I reconsider the way to no

Re: [HACKERS] Online base backup from the hot-standby

2011-06-27 Thread Jun Ishiduka
> Considering everything that has been discussed on this thread so far. > > Do you still think your patch is the best way to accomplish base backups > from standby servers? > If not what changes do you think should be made? I reconsider the way to not use pg_stop_backup(). Process of online bas

Re: [HACKERS] Online base backup from the hot-standby

2011-06-24 Thread Steve Singer
On 11-06-24 12:41 AM, Jun Ishiduka wrote: > > The logic that not use pg_stop_backup() would be difficult, > because pg_stop_backup() is used to identify minRecoveryPoint. > Considering everything that has been discussed on this thread so far. Do you still think your patch is the best way to accom

Re: [HACKERS] Online base backup from the hot-standby

2011-06-23 Thread Jun Ishiduka
> What I think he is proposing would not require using pg_stop_backup() > but you could also modify pg_stop_back() to work as well. > > What do you think of this idea? > > Do you see how the patch can be reworked to accomplish this? The logic that not use pg_stop_backup() would be difficult, be

  1   2   >