Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Robert Haas
On Wed, Sep 8, 2010 at 10:22 AM, Simon Riggs wrote: > On Wed, 2010-09-08 at 09:50 -0400, Robert Haas wrote: > >> So that means we have to make sure that none of the effects of a >> transaction can be seen until WAL is fsync'd on the master AND the >> slave has acked. > > Yes, that's right. And I l

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread David Fetter
On Wed, Sep 08, 2010 at 03:22:46PM +0100, Simon Riggs wrote: > On Wed, 2010-09-08 at 09:50 -0400, Robert Haas wrote: > > > So that means we have to make sure that none of the effects of a > > transaction can be seen until WAL is fsync'd on the master AND the > > slave has acked. > > Yes, that's r

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Simon Riggs
On Wed, 2010-09-08 at 09:50 -0400, Robert Haas wrote: > So that means we have to make sure that none of the effects of a > transaction can be seen until WAL is fsync'd on the master AND the > slave has acked. Yes, that's right. And I like your example; one for the docs. There is a slight complex

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Robert Haas
On Wed, Sep 8, 2010 at 9:32 AM, Fujii Masao wrote: > On Wed, Sep 8, 2010 at 10:07 PM, Robert Haas wrote: >> On Wed, Sep 8, 2010 at 8:30 AM, Fujii Masao wrote:  And in any event, there is ALWAYS a window of time during which the client doesn't know the transaction has committed but

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Fujii Masao
On Wed, Sep 8, 2010 at 10:07 PM, Robert Haas wrote: > On Wed, Sep 8, 2010 at 8:30 AM, Fujii Masao wrote: >>> And in any event, there is ALWAYS a window of >>> time during which the client doesn't know the transaction has >>> committed but other transactions can potentially see its effects. >> >>

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Robert Haas
On Wed, Sep 8, 2010 at 8:30 AM, Fujii Masao wrote: >> And in any event, there is ALWAYS a window of >> time during which the client doesn't know the transaction has >> committed but other transactions can potentially see its effects. > > Yep. The problem here is that synchronous replication is lik

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Simon Riggs
On Wed, 2010-09-08 at 12:04 +0200, Boszormenyi Zoltan wrote: > >> > >> I know why you want that, but I don't think its right. > >> > > > > Agreed. That's why I put the wait before ProcArrayEndTransaction() > > is called. > > > > Then there is no use to implement individual sync/async > rep

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Markus Wanner
On 09/08/2010 12:04 PM, Boszormenyi Zoltan wrote: Then there is no use to implement individual sync/async replicated transactions, period. I disagree. Different transactions have different priorities for latency vs. failure-resistance. An async replicated transaction that waits for a sync r

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Fujii Masao
On Wed, Sep 8, 2010 at 8:43 PM, Robert Haas wrote: > I still don't see why it matters whether you wait before or after > releasing locks.  As soon as the transaction is marked committed in > CLOG, other transactions can potentially see its effects. AFAIR, even if CLOG has been updated, until the

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Robert Haas
On Wed, Sep 8, 2010 at 6:52 AM, Boszormenyi Zoltan wrote: > Fujii Masao írta: >> On Wed, Sep 8, 2010 at 7:04 PM, Boszormenyi Zoltan wrote: >> >>> Then there is no use to implement individual sync/async >>> replicated transactions, period. An async replicated transaction >>> that waits for a sync

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Boszormenyi Zoltan
Fujii Masao írta: > On Wed, Sep 8, 2010 at 7:04 PM, Boszormenyi Zoltan wrote: > >> Then there is no use to implement individual sync/async >> replicated transactions, period. An async replicated transaction >> that waits for a sync replicated transaction because of locks >> will become implicit

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Fujii Masao
On Wed, Sep 8, 2010 at 7:04 PM, Boszormenyi Zoltan wrote: > Then there is no use to implement individual sync/async > replicated transactions, period. An async replicated transaction > that waits for a sync replicated transaction because of locks > will become implicitely sync. It just waits for a

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Boszormenyi Zoltan
Fujii Masao írta: > On Tue, Sep 7, 2010 at 6:02 AM, Simon Riggs wrote: > >> On Mon, 2010-09-06 at 22:32 +0200, Boszormenyi Zoltan wrote: >> >>> (in commit) >>> write wal record >>> release locks/etc >> wait for sync ack >>> >>> In the first case, the contention is obviously increased. >>

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Fujii Masao
On Tue, Sep 7, 2010 at 6:02 AM, Simon Riggs wrote: > On Mon, 2010-09-06 at 22:32 +0200, Boszormenyi Zoltan wrote: >> (in commit) >> write wal record >> release locks/etc   > wait for sync ack >> >> In the first case, the contention is obviously increased. >> With this, we are creating more idle ti

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 4:06 PM, marcin mank wrote: > On Tue, Sep 7, 2010 at 5:17 PM, Tom Lane wrote: >> We can *not* allow the slave to replay WAL ahead of what is known >> committed to disk on the master.  The only way to make that safe >> is the compare-notes-and-ship-WAL-back approach that Rob

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread marcin mank
On Tue, Sep 7, 2010 at 5:17 PM, Tom Lane wrote: > We can *not* allow the slave to replay WAL ahead of what is known > committed to disk on the master.  The only way to make that safe > is the compare-notes-and-ship-WAL-back approach that Robert mentioned. > > If you feel that decoupling WAL applic

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
On 09/07/2010 05:55 PM, Markus Wanner wrote: Robert's argument Sorry, I meant Ron. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
Hi, On 09/07/2010 06:00 PM, Robert Haas wrote: People who are more concerned about performance than robustness aren't going to use sync rep in the first place. I'm advocating sync (or eager, FWIW) replication for years, now. One of the hardest preconception I'm always confronted with is: this

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 11:06 AM, Simon Riggs wrote: > On Tue, 2010-09-07 at 16:31 +0200, Markus Wanner wrote: >> On 09/07/2010 04:15 PM, Robert Haas wrote: >> > In theory, that's true, but if we do that, then there's an even bigger >> > problem: the slave might have replayed WAL ahead of the maste

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
Hi, On 09/07/2010 05:17 PM, Tom Lane wrote: Oh yes it is. If the slave replays WAL that didn't happen on the master, it might for instance have heap tuples in TID slots that are empty on the master, or index pages laid out differently from the master. Trying to apply additional WAL from the ma

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Tom Lane
Simon Riggs writes: > On Tue, 2010-09-07 at 11:17 -0400, Tom Lane wrote: >> We can *not* allow the slave to replay WAL ahead of what is known >> committed to disk on the master. The only way to make that safe >> is the compare-notes-and-ship-WAL-back approach that Robert mentioned. >> >> If you

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
On 09/07/2010 04:47 PM, Ron Mayer wrote: In that situation, wouldn't it be possible that a different client queried the slave and already saw the result of that transaction which would later be rolled back? Good point, yes. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 11:17 -0400, Tom Lane wrote: > Markus Wanner writes: > > On 09/07/2010 04:15 PM, Robert Haas wrote: > >> In theory, that's true, but if we do that, then there's an even bigger > >> problem: the slave might have replayed WAL ahead of the master > >> location; therefore the sla

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Tom Lane
Markus Wanner writes: > On 09/07/2010 04:15 PM, Robert Haas wrote: >> In theory, that's true, but if we do that, then there's an even bigger >> problem: the slave might have replayed WAL ahead of the master >> location; therefore the slave is now corrupt and a new base backup >> must be taken. >

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 16:31 +0200, Markus Wanner wrote: > On 09/07/2010 04:15 PM, Robert Haas wrote: > > In theory, that's true, but if we do that, then there's an even bigger > > problem: the slave might have replayed WAL ahead of the master > > location; therefore the slave is now corrupt and a n

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Ron Mayer
Markus Wanner wrote: > On 09/07/2010 02:16 PM, Robert Haas wrote: >> practice, this means that the master and standby need to compare notes >> on the ending WAL location and whichever one is further advanced needs >> to stream the intervening records to the other. > > Not necessarily, no. Remember

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be taken. The slave isn't corrupt. It would su

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 9:45 AM, Markus Wanner wrote: > On 09/07/2010 02:16 PM, Robert Haas wrote: >> >> Right, definitely.  The trouble is that if they happen concurrently, >> and there's a crash, you have to be prepared for the possibility that >> either one of the two has completed and the other

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
On 09/07/2010 02:16 PM, Robert Haas wrote: Right, definitely. The trouble is that if they happen concurrently, and there's a crash, you have to be prepared for the possibility that either one of the two has completed and the other is not. Understood. In practice, this means that the master a

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 4:01 AM, Markus Wanner wrote: > In any case, a server failure in between the commit request of the client > and the commit confirmation for the client results in a client that can't > tell if its transaction committed or not. > > So why do we care what to do first internally

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
Hi, On 05/27/2010 01:28 PM, Robert Haas wrote: How do you propose to guarantee that? ISTM that you have to either commit locally first, or send the commit to the remote first. Either way, the two events won't occur exactly simultaneously. I'm not getting the point of this discussion. As long

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Robert Haas
On Mon, Sep 6, 2010 at 5:02 PM, Simon Riggs wrote: > Then I respectfully suggest that you're releasing locks too early. > > Your proposal would allow a 2nd user to see the results of the 1st > user's transaction before the 1st user knew about whether it had > committed or not. Marking the transac

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 16:11 -0400, Tom Lane wrote: > Simon Riggs writes: > > On Mon, 2010-09-06 at 21:45 +0200, Boszormenyi Zoltan wrote: > >> The locks held by a transaction are released after > >> RecordTransactionCommit(), and waiting for the sync ack > >> happens in this function. Now what hap

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 23:07 +0100, Greg Stark wrote: > On Mon, Sep 6, 2010 at 10:02 PM, Simon Riggs wrote: > > Then I respectfully suggest that you're releasing locks too early. > > > > Your proposal would allow a 2nd user to see the results of the 1st > > user's transaction before the 1st user kn

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Greg Stark
On Mon, Sep 6, 2010 at 10:02 PM, Simon Riggs wrote: > Then I respectfully suggest that you're releasing locks too early. > > Your proposal would allow a 2nd user to see the results of the 1st > user's transaction before the 1st user knew about whether it had > committed or not. > > I know why you

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 22:32 +0200, Boszormenyi Zoltan wrote: > (in commit) > write wal record > release locks/etcwait for sync ack > > In the first case, the contention is obviously increased. > With this, we are creating more idle time in the server > instead of letting other transactions do

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Boszormenyi Zoltan
Simon Riggs írta: > On Mon, 2010-09-06 at 21:45 +0200, Boszormenyi Zoltan wrote: > >> Dimitri Fontaine írta: >> >>> Boszormenyi Zoltan writes: >>> >>> Sorry for answering such an old mail, but what is the purpose of a transaction level synchronous behaviour if async tra

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Tom Lane
Simon Riggs writes: > On Mon, 2010-09-06 at 21:45 +0200, Boszormenyi Zoltan wrote: >> The locks held by a transaction are released after >> RecordTransactionCommit(), and waiting for the sync ack >> happens in this function. Now what happens when a sync >> transaction hold a lock that an async one

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 21:45 +0200, Boszormenyi Zoltan wrote: > Dimitri Fontaine írta: > > Boszormenyi Zoltan writes: > > > >> Sorry for answering such an old mail, but what is the purpose of > >> a transaction level synchronous behaviour if async transactions > >> can be held back by a sync tra

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Boszormenyi Zoltan
Dimitri Fontaine írta: > Boszormenyi Zoltan writes: > >> Sorry for answering such an old mail, but what is the purpose of >> a transaction level synchronous behaviour if async transactions >> can be held back by a sync transaction? >> > > I don't understand why it would be the case (sync h

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Dimitri Fontaine
Boszormenyi Zoltan writes: > Sorry for answering such an old mail, but what is the purpose of > a transaction level synchronous behaviour if async transactions > can be held back by a sync transaction? I don't understand why it would be the case (sync holding back async transactions) — it's been

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Boszormenyi Zoltan
Fujii Masao írta: > On Fri, Sep 3, 2010 at 6:43 PM, Boszormenyi Zoltan wrote: > >> In my patch, when the transactions were waiting for ack from >> the standby, they have already released all their locks, the wait >> happened at the latest possible point in CommitTransaction(). >> >> In Fujii's

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Fujii Masao
On Fri, Sep 3, 2010 at 6:43 PM, Boszormenyi Zoltan wrote: > In my patch, when the transactions were waiting for ack from > the standby, they have already released all their locks, the wait > happened at the latest possible point in CommitTransaction(). > > In Fujii's patch (I am looking at synch_r

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Boszormenyi Zoltan
Hi, Dimitri Fontaine írta: > Simon Riggs writes: > >> Seems strange. If you have 2 standbys and you say you would like node1 >> to be the preferred candidate, then you load it so heavily that a remote >> server with by-definition much larger network delay responds first, then >> I say your pre

Re: [HACKERS] Synchronization levels in SR

2010-06-04 Thread Jan Wieck
On 6/4/2010 4:22 PM, Robert Haas wrote: On Fri, Jun 4, 2010 at 3:35 PM, Jan Wieck wrote: So that justifies adding code, that the community needs to maintain and document, to the core system. If only I could find some monitoring case for transaction commit orders ... sigh! Dude, I'm not the on

Re: [HACKERS] Synchronization levels in SR

2010-06-04 Thread Robert Haas
On Fri, Jun 4, 2010 at 3:35 PM, Jan Wieck wrote: > On 6/3/2010 10:57 PM, Robert Haas wrote: >> >> On Thu, Jun 3, 2010 at 8:47 PM, Jan Wieck wrote: >>> >>> On 5/27/2010 4:31 PM, Bruce Momjian wrote: Also, what would be cool would be if you could run a query on the master to view the

Re: [HACKERS] Synchronization levels in SR

2010-06-04 Thread Jan Wieck
On 6/3/2010 10:57 PM, Robert Haas wrote: On Thu, Jun 3, 2010 at 8:47 PM, Jan Wieck wrote: On 5/27/2010 4:31 PM, Bruce Momjian wrote: Also, what would be cool would be if you could run a query on the master to view the SR commit mode of each slave. What would be the use case for such a query?

Re: [HACKERS] Synchronization levels in SR

2010-06-04 Thread David Fetter
On Thu, Jun 03, 2010 at 10:57:05PM -0400, Robert Haas wrote: > On Thu, Jun 3, 2010 at 8:47 PM, Jan Wieck wrote: > > What would be the use case for such a query? > > Monitoring? s/\?/!/; Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter

Re: [HACKERS] Synchronization levels in SR

2010-06-03 Thread Robert Haas
On Thu, Jun 3, 2010 at 8:47 PM, Jan Wieck wrote: > On 5/27/2010 4:31 PM, Bruce Momjian wrote: >> >> Heikki Linnakangas wrote: >>> >>> BTW, I think we're going to need a separate config file for listing the >>> standbys anyway. There you can write per-server rules and options, but >>> explicitly kn

Re: [HACKERS] Synchronization levels in SR

2010-06-03 Thread Jan Wieck
On 5/27/2010 4:31 PM, Bruce Momjian wrote: Heikki Linnakangas wrote: BTW, I think we're going to need a separate config file for listing the standbys anyway. There you can write per-server rules and options, but explicitly knowing about all the standbys also allows the master to recycle WAL as

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Greg Smith
Tom Lane wrote: Heikki Linnakangas writes: It's pretty scary to call a user-defined function at that point in transaction. Not so much "pretty scary" as "zero chance of being accepted". And I do mean zero. I swear, you guys are such buzzkills some days. I was suggesting a model

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Tom Lane
Heikki Linnakangas writes: > It's pretty scary to call a user-defined function at that point in > transaction. Not so much "pretty scary" as "zero chance of being accepted". And I do mean zero. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@pos

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Simon Riggs
On Wed, 2010-06-02 at 03:22 -0400, Greg Smith wrote: > Heikki Linnakangas wrote: > > The possibilities are endless... Your proposal above covers a pretty > > good set of scenarios, but it's by no means complete. If we try to > > solve everything the configuration will need to be written in a > >

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Heikki Linnakangas
On 02/06/10 10:22, Greg Smith wrote: Heikki Linnakangas wrote: The possibilities are endless... Your proposal above covers a pretty good set of scenarios, but it's by no means complete. If we try to solve everything the configuration will need to be written in a Turing-complete Replication Descr

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Greg Smith
Heikki Linnakangas wrote: The possibilities are endless... Your proposal above covers a pretty good set of scenarios, but it's by no means complete. If we try to solve everything the configuration will need to be written in a Turing-complete Replication Description Language. We'll have to pick

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Robert Haas
On May 27, 2010, at 4:31 PM, Bruce Momjian wrote: Heikki Linnakangas wrote: BTW, I think we're going to need a separate config file for listing the standbys anyway. There you can write per-server rules and options, but explicitly knowing about all the standbys also allows the master to recy

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Bruce Momjian
Heikki Linnakangas wrote: > BTW, I think we're going to need a separate config file for listing the > standbys anyway. There you can write per-server rules and options, but > explicitly knowing about all the standbys also allows the master to > recycle WAL as soon as it has been streamed to all

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Bruce Momjian
Simon Riggs wrote: > On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: > > > I guess that dropping the support of #3 doesn't reduce complexity > > since the code of #3 is almost the same as that of #2. Like > > walreceiver sends the ACK after receiving the WAL in #2 case, it has > > only to do

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Robert Haas
On Thu, May 27, 2010 at 9:09 AM, Fujii Masao wrote: > On Thu, May 27, 2010 at 9:48 PM, Robert Haas wrote: >> There could still be additional transactions that the original master >> has committed locally but were not acked to the client.  I guess you'd >> just work around that by taking a new bas

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 9:48 PM, Robert Haas wrote: > There could still be additional transactions that the original master > has committed locally but were not acked to the client.  I guess you'd > just work around that by taking a new base backup from the new master. Right. Unfortunately the t

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Robert Haas
On Thu, May 27, 2010 at 8:02 AM, Fujii Masao wrote: > On Thu, May 27, 2010 at 8:28 PM, Robert Haas wrote: >> On Thu, May 27, 2010 at 3:13 AM, Fujii Masao wrote: >>> (1) most standard case: 1 master + 1 "sync" standby (near) >>>    When the master goes down, something like a clusterware detects t

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 8:30 PM, Simon Riggs wrote: > There already is a first patch to the community that implements quorum > commit, just not by you. Yeah, AFAIK, that patch includes also per-standby setting. > If you have a better way, describe it in detail and in full now, with > reference t

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 8:30 PM, Simon Riggs wrote: > Why does it? I just explained a design where that wasn't required. Hmm.. my expression might have been ambiguous. Walreceiver still needs to wait for WAL flush by walwriter *before* sending the ACK to the master, in #3 case. Because, in #3, th

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 8:28 PM, Robert Haas wrote: > On Thu, May 27, 2010 at 3:13 AM, Fujii Masao wrote: >> (1) most standard case: 1 master + 1 "sync" standby (near) >>    When the master goes down, something like a clusterware detects that >>    failure, and brings the standby online. Since we

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 19:50 +0900, Fujii Masao wrote: > For now, I agree that we support a quorum commit feature for 9.1 or later. > But I don't think that it's simpler, more intuitive and easier-to-understand > than per-standby setting. So I think that we should include the per-standby > setting

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 20:13 +0900, Fujii Masao wrote: > On Thu, May 27, 2010 at 7:33 PM, Simon Riggs wrote: > > On Thu, 2010-05-27 at 19:21 +0900, Fujii Masao wrote: > >> On Thu, May 27, 2010 at 6:30 PM, Simon Riggs wrote: > > > >> > Would it be easier to have > >> > this happen in a second pair

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Robert Haas
On Thu, May 27, 2010 at 3:13 AM, Fujii Masao wrote: > (1) most standard case: 1 master + 1 "sync" standby (near) >    When the master goes down, something like a clusterware detects that >    failure, and brings the standby online. Since we can ensure that the >    standby has all the committed tr

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Dimitri Fontaine
Simon Riggs writes: > Seems strange. If you have 2 standbys and you say you would like node1 > to be the preferred candidate, then you load it so heavily that a remote > server with by-definition much larger network delay responds first, then > I say your preference was wrong. There's a communica

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 7:33 PM, Simon Riggs wrote: > On Thu, 2010-05-27 at 19:21 +0900, Fujii Masao wrote: >> On Thu, May 27, 2010 at 6:30 PM, Simon Riggs wrote: > >> > Would it be easier to have >> > this happen in a second pair of processes WALSynchroniser (on primary) >> > and WAL Acknowledge

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 7:21 PM, Simon Riggs wrote: > Seems strange. If you have 2 standbys and you say you would like node1 > to be the preferred candidate, then you load it so heavily that a remote > server with by-definition much larger network delay responds first, then > I say your preference

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 19:21 +0900, Fujii Masao wrote: > On Thu, May 27, 2010 at 6:30 PM, Simon Riggs wrote: > > Would it be easier to have > > this happen in a second pair of processes WALSynchroniser (on primary) > > and WAL Acknowledger (on standby). WALAcknowledger would send back a > > stream

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 6:30 PM, Simon Riggs wrote: > Who sends the ack message? walreceiver > Who receives it? walsender > Would it be easier to have > this happen in a second pair of processes WALSynchroniser (on primary) > and WAL Acknowledger (on standby). WALAcknowledger would send back a

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 16:13 +0900, Fujii Masao wrote: > On Wed, May 26, 2010 at 10:37 PM, Simon Riggs wrote: > > If the remote server responded first, then that proves it is a better > > candidate for failover than the one you think of as near. If the two > > standbys vary over time then you have

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 16:13 +0900, Fujii Masao wrote: > On Wed, May 26, 2010 at 10:37 PM, Simon Riggs wrote: > > Please explain what will happen when the near server is unavailable, > > with per-standby settings. Please also explain what will happen if we > > choose to have 4 or 5 servers to main

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 16:35 +0900, Fujii Masao wrote: > On Thu, May 27, 2010 at 3:21 PM, Simon Riggs wrote: > > On Thu, 2010-05-27 at 11:28 +0900, Fujii Masao wrote: > >> On Wed, May 26, 2010 at 10:20 PM, Simon Riggs > >> wrote: > >> > On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: > >> >

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 10:09 +0300, Heikki Linnakangas wrote: > No, not necessarily. As I said above, you might just want a guarantee > that *if* you query the standby, you get up-to-date results. Of course. COMMIT was already one of the options, so this comment was already understood. What we ar

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Dimitri Fontaine
Heikki Linnakangas writes: > On 26/05/10 23:31, Dimitri Fontaine wrote: >> So if you want simplicity to admin, effective data availability and >> precise control over the global setup, I say go for: >> a. transaction level control of the replication level >> b. cascading support >> c. quorum

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 3:21 PM, Simon Riggs wrote: > On Thu, 2010-05-27 at 11:28 +0900, Fujii Masao wrote: >> On Wed, May 26, 2010 at 10:20 PM, Simon Riggs wrote: >> > On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: >> > >> >> I guess that dropping the support of #3 doesn't reduce complexi

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Wed, May 26, 2010 at 10:37 PM, Simon Riggs wrote: > If the remote server responded first, then that proves it is a better > candidate for failover than the one you think of as near. If the two > standbys vary over time then you have network problems that will > directly affect the performance o

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Heikki Linnakangas
On 27/05/10 09:51, Simon Riggs wrote: On Thu, 2010-05-27 at 02:18 +0300, Heikki Linnakangas wrote: In practice, hard synchronous "don't return ever until the commit hits the standby" behavior is rarely what admins actually want, because it's disastrous from an availability point of view. More li

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Thu, 2010-05-27 at 02:18 +0300, Heikki Linnakangas wrote: > On 27/05/10 01:23, Simon Riggs wrote: > > On Thu, 2010-05-27 at 00:21 +0300, Heikki Linnakangas wrote: > >> On 26/05/10 23:31, Dimitri Fontaine wrote: > >>>d. choice of commit or rollback at timeout > >> > >> Rollback is not an opti

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Thu, 2010-05-27 at 11:28 +0900, Fujii Masao wrote: > On Wed, May 26, 2010 at 10:20 PM, Simon Riggs wrote: > > On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: > > > >> I guess that dropping the support of #3 doesn't reduce complexity > >> since the code of #3 is almost the same as that of

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Fujii Masao
On Wed, May 26, 2010 at 10:20 PM, Simon Riggs wrote: > On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: > >> I guess that dropping the support of #3 doesn't reduce complexity >> since the code of #3 is almost the same as that of #2. Like >> walreceiver sends the ACK after receiving the WAL in

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Heikki Linnakangas
On 27/05/10 01:23, Simon Riggs wrote: On Thu, 2010-05-27 at 00:21 +0300, Heikki Linnakangas wrote: On 26/05/10 23:31, Dimitri Fontaine wrote: d. choice of commit or rollback at timeout Rollback is not an option. There is no going back after the commit record has been flushed to disk or sen

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Thu, 2010-05-27 at 00:21 +0300, Heikki Linnakangas wrote: > On 26/05/10 23:31, Dimitri Fontaine wrote: > > d. choice of commit or rollback at timeout > > Rollback is not an option. There is no going back after the commit > record has been flushed to disk or sent to a standby. There's defini

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 17:31 -0400, Jan Wieck wrote: > You can do this only with per standby options, by giving each standby a > weight, or a number of votes. Your DEV server would have a weight of > zero, while your production standby's have higher weights, depending on > their importance for y

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Heikki Linnakangas
On 26/05/10 23:31, Dimitri Fontaine wrote: So if you want simplicity to admin, effective data availability and precise control over the global setup, I say go for: a. transaction level control of the replication level b. cascading support c. quorum with timeout d. choice of commit or roll

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Jan Wieck
On 5/26/2010 12:55 PM, Heikki Linnakangas wrote: On 26/05/10 18:31, Robert Haas wrote: And frankly, I don't think it's possible for quorum commit to reduce the number of parameters. Even if we have that feature available, not everyone will want to use it. And the people who don't will presumab

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Heikki Linnakangas
On 26/05/10 23:31, Dimitri Fontaine wrote: d. choice of commit or rollback at timeout Rollback is not an option. There is no going back after the commit record has been flushed to disk or sent to a standby. The choice is to either commit anyway after the timeout, or wait forever. -- Hei

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Dimitri Fontaine
Simon Riggs writes: > On Wed, 2010-05-26 at 19:55 +0300, Heikki Linnakangas wrote: >> Now you want to set up a temporary replica of the master at a >> development server, for testing purposes. If you set quorum to 2, your >> development server becomes critical infrastructure, which is not what >

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Joshua D. Drake
On Wed, 2010-05-26 at 15:37 -0400, Robert Haas wrote: > On Wed, May 26, 2010 at 3:13 PM, Simon Riggs wrote: > >> I don't really understand this comment. I have said, and I believe, > >> that a system without quorum commit is simpler than one with quorum > >> commit. I'd debate the point with you

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Robert Haas
On Wed, May 26, 2010 at 3:13 PM, Simon Riggs wrote: >> I don't really understand this comment.  I have said, and I believe, >> that a system without quorum commit is simpler than one with quorum >> commit.  I'd debate the point with you but I find the point so >> self-evident that I don't even kno

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 14:30 -0400, Robert Haas wrote: > On Wed, May 26, 2010 at 1:26 PM, Simon Riggs wrote: > > On Wed, 2010-05-26 at 11:31 -0400, Robert Haas wrote: > >> > Your reply has again avoided the subject of how we would handle failure > >> > modes with per-standby settings. That is impor

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Robert Haas
On Wed, May 26, 2010 at 1:26 PM, Simon Riggs wrote: > On Wed, 2010-05-26 at 11:31 -0400, Robert Haas wrote: >> > Your reply has again avoided the subject of how we would handle failure >> > modes with per-standby settings. That is important. >> >> I don't think anyone is avoiding that, we just hav

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Heikki Linnakangas
On 26/05/10 20:40, Simon Riggs wrote: On Wed, 2010-05-26 at 19:55 +0300, Heikki Linnakangas wrote: If you set quorum to 1, it also becomes critical infrastructure, because it's possible that a transaction has been replicated to the test server but not the real production standby, and a meteor st

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Kevin Grittner
Heikki Linnakangas wrote: >> Unless we have a transaction manager and do proper distributed >> transactions, how do you avoid edge conditions like that? > > Yeah, I guess you can't. You can guarantee that a commit is > always safely flushed first in the master, or in the standby, but > without

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Heikki Linnakangas
On 26/05/10 20:33, Kevin Grittner wrote: Heikki Linnakangas wrote: Although, if the master crashes at that point, and quickly recovers, you could see the last transactions committed on the master before they're replicated to the standby. Versus having the transaction committed on one or more

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Robert Haas
On Wed, May 26, 2010 at 1:24 PM, Heikki Linnakangas wrote: > On 26/05/10 20:10, Kevin Grittner wrote: >> >> Heikki Linnakangas  wrote: >> >>> One way to do that would be to refrain from flushing the commit >>> record to disk on the master until the standby has acknowledged >>> it. >> >> I'm not cl

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 19:55 +0300, Heikki Linnakangas wrote: > Now you want to set up a temporary replica of the master at a > development server, for testing purposes. If you set quorum to 2, your > development server becomes critical infrastructure, which is not what > you want. That's a good

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Kevin Grittner
Heikki Linnakangas wrote: > Although, if the master crashes at that point, and quickly > recovers, you could see the last transactions committed on the > master before they're replicated to the standby. Versus having the transaction committed on one or more slaves but not on the master? Unles

  1   2   >