Re: [HACKERS] Streaming replication, and walsender during recovery

2010-03-16 Thread Fujii Masao
On Tue, Mar 16, 2010 at 6:11 PM, Heikki Linnakangas wrote: > Commmitted. I edited the message and error code a bit: > > ereport(FATAL, >        (errcode(ERRCODE_CANNOT_CONNECT_NOW), >         errmsg("recovery is still in progress, can't accept WAL > streaming connections"))); > > ERRCODE_CANNOT_CO

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-03-16 Thread Heikki Linnakangas
Fujii Masao wrote: > On Mon, Jan 18, 2010 at 2:19 PM, Fujii Masao wrote: >> When I configured a cascaded standby (i.e, made the additional >> standby server connect to the standby), I got the following >> errors, and a cascaded standby didn't start replication. >> >> ERROR: timeline 0 of the pri

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-02-17 Thread Fujii Masao
On Mon, Jan 18, 2010 at 2:19 PM, Fujii Masao wrote: > When I configured a cascaded standby (i.e, made the additional > standby server connect to the standby), I got the following > errors, and a cascaded standby didn't start replication. > >  ERROR:  timeline 0 of the primary does not match recove

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-29 Thread Fujii Masao
On Fri, Jan 29, 2010 at 5:41 PM, Simon Riggs wrote: >> To improve the situation, I think that we need to use >> checkpoint_segment/timeout as a trigger of restartpoint, regardless >> of the checkpoint record. Though I'm not sure that is possible and >> should be included in v9.0. > > Yes, that is

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 17:31 +0900, Fujii Masao wrote: > On Fri, Jan 29, 2010 at 4:13 AM, Simon Riggs wrote: > > Hmm, I'm sorry but that's bogus. Retaining so much WAL that we are > > strongly in danger of blowing disk space is not what I would call a > > safety feature. Since there is no way to co

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-29 Thread Fujii Masao
On Fri, Jan 29, 2010 at 4:13 AM, Simon Riggs wrote: > Hmm, I'm sorry but that's bogus. Retaining so much WAL that we are > strongly in danger of blowing disk space is not what I would call a > safety feature. Since there is no way to control or restrain the number > of files for certain, that appr

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 09:49 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Thu, 2010-01-28 at 21:00 +0200, Heikki Linnakangas wrote: > >> I think it is a pretty important safety feature that we keep all the > >> WAL around that's needed to recover the standby. To avoid > >> out-of-dis

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: > On Thu, 2010-01-28 at 21:00 +0200, Heikki Linnakangas wrote: >> I think it is a pretty important safety feature that we keep all the >> WAL around that's needed to recover the standby. To avoid >> out-of-disk-space situation, it's probably enough in practice to set >> checkpoin

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Josh Berkus
Guys, > Hmm, I'm sorry but that's bogus. Retaining so much WAL that we are > strongly in danger of blowing disk space is not what I would call a > safety feature. Since there is no way to control or restrain the number > of files for certain, that approach seems fatally flawed. Reducing > checkpoi

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Joshua D. Drake wrote: > ...if with SR the entire log must be written before it streams to the slaves. No. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 21:00 +0200, Heikki Linnakangas wrote: > However, since not every checkpoint is a restartpoint we might easily > > end up with significantly more WAL files on the standby than would > > normally be there when it would be a primary. Not sure if that is an > > issue in this case

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: > On Thu, 2010-01-28 at 10:48 -0500, Tom Lane wrote: >> Fujii Masao writes: >>> How about just making a restore_command copy the WAL files as the >>> normal one (e.g., ...) instead of a pg_xlog/RECOVERYXLOG? >>> Though we need to worry about deleting them, we can easily leav

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 20:49 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > I'm a little worried the feature set of streaming rep isn't any better > > than what we have already. > > Huh? Are you thinking of the "Record-based Log Shipping" described in > the manual, using a program to pol

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: > I'm a little worried the feature set of streaming rep isn't any better > than what we have already. Huh? Are you thinking of the "Record-based Log Shipping" described in the manual, using a program to poll pg_xlogfile_name_offset() in a tight loop, as a replacement for streami

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Joshua D. Drake
On Thu, 2010-01-28 at 13:05 -0500, Tom Lane wrote: > > I'm a little worried the feature set of streaming rep isn't any better > > than what we have already. > > Nonsense. Getting rid of the WAL-segment-based shipping delays is a > quantum improvement --- it means we actually have something appro

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Tom Lane wrote: > Fujii Masao writes: >> How about just making a restore_command copy the WAL files as the >> normal one (e.g., ...) instead of a pg_xlog/RECOVERYXLOG? >> Though we need to worry about deleting them, we can easily leave >> the task to the bgwriter. > > The reason for doing it

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Greg Smith
Tom Lane wrote: (Anyway, the argument that it's important for performance is pure speculation AFAIK, untainted by any actual measurements. Given the lack of optimization of WAL replay, it seems entirely possible that the last thing you want to burden a slave with is sourcing data to more slaves.

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 13:05 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Thu, 2010-01-28 at 11:41 -0500, Tom Lane wrote: > >> FWIW, I don't agree with that prioritization in the least. Cascading > >> is something we could leave till 9.1, or even later, and > > > Not what you said just a

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Tom Lane
Simon Riggs writes: > On Thu, 2010-01-28 at 11:41 -0500, Tom Lane wrote: >> FWIW, I don't agree with that prioritization in the least. Cascading >> is something we could leave till 9.1, or even later, and > Not what you said just a few days ago. Me? I don't recall having said a word about cas

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 11:41 -0500, Tom Lane wrote: > Simon Riggs writes: > > I'm keen to allow cascading in 9.0. If you pull both synch rep and > > cascading you're not offering much that isn't already there. > > FWIW, I don't agree with that prioritization in the least. Cascading > is something

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 12:09 -0500, Robert Haas wrote: > I agree. According to > http://wiki.postgresql.org/wiki/Hot_Standby_TODO , the only must-fix > issues that remain prior to beta are (1) implementing the new VACUUM > FULL for system relations, and (2) some documentation improvements. > It's

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Robert Haas
On Thu, Jan 28, 2010 at 11:41 AM, Tom Lane wrote: > Simon Riggs writes: >> I'm keen to allow cascading in 9.0. If you pull both synch rep and >> cascading you're not offering much that isn't already there. > > FWIW, I don't agree with that prioritization in the least.  Cascading > is something we

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Tom Lane
Simon Riggs writes: > I'm keen to allow cascading in 9.0. If you pull both synch rep and > cascading you're not offering much that isn't already there. FWIW, I don't agree with that prioritization in the least. Cascading is something we could leave till 9.1, or even later, and hardly anyone woul

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 10:48 -0500, Tom Lane wrote: > Fujii Masao writes: > > How about just making a restore_command copy the WAL files as the > > normal one (e.g., ...) instead of a pg_xlog/RECOVERYXLOG? > > Though we need to worry about deleting them, we can easily leave > > the task to the

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Tom Lane
Fujii Masao writes: > How about just making a restore_command copy the WAL files as the > normal one (e.g., ...) instead of a pg_xlog/RECOVERYXLOG? > Though we need to worry about deleting them, we can easily leave > the task to the bgwriter. The reason for doing it that way was to limit disk

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Fujii Masao
On Thu, Jan 28, 2010 at 7:43 PM, Heikki Linnakangas wrote: > How about restoring/streaming files to a new directory, say > pg_xlog/restored/, with the real filenames? At least in standby_mode, > probably best to keep the current behavior in PITR. That would feel more > clean, you could easily tell

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Fujii Masao wrote: > On Thu, Jan 28, 2010 at 4:47 PM, Heikki Linnakangas > wrote: >> I think there's a race condition at the end of recovery. When the >> shutdown checkpoint is written, with new TLI, doesn't a cascading >> walsender try to send that to the standby as soon as it's flushed to >> dis

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Fujii Masao
On Thu, Jan 28, 2010 at 4:47 PM, Heikki Linnakangas wrote: > I think there's a race condition at the end of recovery. When the > shutdown checkpoint is written, with new TLI, doesn't a cascading > walsender try to send that to the standby as soon as it's flushed to > disk? But it won't find it in

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-27 Thread Heikki Linnakangas
Fujii Masao wrote: > OK. Here is the patch which supports a walsender process during recovery; > > * Change walsender so as to send the WAL written by the walreceiver > if it has been started during recovery. > * Kill the walsenders started during recovery at the end of recovery > because repl

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-20 Thread Fujii Masao
On Tue, Jan 19, 2010 at 4:41 PM, Simon Riggs wrote: > It would be useful to have the ps display differentiate between multiple > walsenders, and in this case have it indicate cascading also. Since a normal walsender and a "cascading" one will not be running at the same time, I don't think that it

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-18 Thread Simon Riggs
On Tue, 2010-01-19 at 15:04 +0900, Fujii Masao wrote: > > > > There is an ERROR, but no problem AFAICS. The tli isn't set until end of > > recovery because it doesn't need to have been set yet. That shouldn't > > prevent retrieving WAL data. > > OK. Here is the patch which supports a walsender pro

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-18 Thread Fujii Masao
On Mon, Jan 18, 2010 at 11:42 PM, Simon Riggs wrote: > On Mon, 2010-01-18 at 09:31 -0500, Tom Lane wrote: >> Fujii Masao writes: >> > When I configured a cascaded standby (i.e, made the additional >> > standby server connect to the standby), I got the following >> > errors, and a cascaded standby

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-18 Thread Simon Riggs
On Mon, 2010-01-18 at 09:31 -0500, Tom Lane wrote: > Fujii Masao writes: > > When I configured a cascaded standby (i.e, made the additional > > standby server connect to the standby), I got the following > > errors, and a cascaded standby didn't start replication. > > > ERROR: timeline 0 of th

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-18 Thread Tom Lane
Fujii Masao writes: > When I configured a cascaded standby (i.e, made the additional > standby server connect to the standby), I got the following > errors, and a cascaded standby didn't start replication. > ERROR: timeline 0 of the primary does not match recovery target timeline 1 > I didn't

[HACKERS] Streaming replication, and walsender during recovery

2010-01-17 Thread Fujii Masao
Hi, When I configured a cascaded standby (i.e, made the additional standby server connect to the standby), I got the following errors, and a cascaded standby didn't start replication. ERROR: timeline 0 of the primary does not match recovery target timeline 1 I didn't care about that case so f