Re: [HACKERS] Replication identifiers, take 3

2014-10-29 Thread Simon Riggs
On 2 October 2014 09:49, Heikki Linnakangas wrote: >> What I've previously suggested (and which works well in BDR) is to add >> the internal id to the XLogRecord struct. There's 2 free bytes of >> padding that can be used for that purpose. > > > Adding a field to XLogRecord for this feels wrong.

Re: [HACKERS] Replication identifiers, take 3

2014-10-04 Thread Jim Nasby
On 10/2/14, 7:28 AM, Robert Haas wrote: On Thu, Oct 2, 2014 at 4:49 AM, Heikki Linnakangas wrote: >An origin column in the table itself helps tremendously to debug issues with >the replication system. In many if not most scenarios, I think you'd want to >have that extra column, even if it's no

Re: [HACKERS] Replication identifiers, take 3

2014-10-02 Thread Robert Haas
On Thu, Oct 2, 2014 at 4:49 AM, Heikki Linnakangas wrote: > An origin column in the table itself helps tremendously to debug issues with > the replication system. In many if not most scenarios, I think you'd want to > have that extra column, even if it's not strictly required. I like a lot of wha

Re: [HACKERS] Replication identifiers, take 3

2014-10-02 Thread Andres Freund
On 2014-10-02 11:49:31 +0300, Heikki Linnakangas wrote: > On 09/23/2014 09:24 PM, Andres Freund wrote: > >I've previously started two threads about replication identifiers. Check > >http://archives.postgresql.org/message-id/20131114172632.GE7522%40alap2.anarazel.de > >and > >http://archives.postgre

Re: [HACKERS] Replication identifiers, take 3

2014-10-02 Thread Heikki Linnakangas
On 09/23/2014 09:24 PM, Andres Freund wrote: I've previously started two threads about replication identifiers. Check http://archives.postgresql.org/message-id/20131114172632.GE7522%40alap2.anarazel.de and http://archives.postgresql.org/message-id/20131211153833.GB25227%40awork2.anarazel.de . Th

Re: [HACKERS] Replication identifiers, take 3

2014-09-29 Thread Robert Haas
On Sat, Sep 27, 2014 at 12:11 PM, Steve Singer wrote: > If we were now increasing the WAL record size anyway for some unrelated > reason, would we be willing to increase it by a further 2 bytes for the node > identifier? Obviously not. Otherwise Andres would be proposing to put an OID in there i

Re: [HACKERS] Replication identifiers, take 3

2014-09-29 Thread Andres Freund
On 2014-09-27 12:11:16 -0400, Steve Singer wrote: > On 09/26/2014 06:05 PM, Andres Freund wrote: > >On 2014-09-26 14:57:12 -0400, Robert Haas wrote: > >Sure, it'll possibly not be trivial to move them elsewhere. On the other > >hand, the padding bytes have been unused for 8+ years without somebody

Re: [HACKERS] Replication identifiers, take 3

2014-09-27 Thread Steve Singer
On 09/26/2014 10:21 AM, Andres Freund wrote: On 2014-09-26 09:53:09 -0400, Robert Haas wrote: On Fri, Sep 26, 2014 at 5:05 AM, Andres Freund wrote: Let me try to summarize the information requirements for each of these things. For #1, you need to know, after crash recovery, for each standby,

Re: [HACKERS] Replication identifiers, take 3

2014-09-27 Thread Steve Singer
On 09/26/2014 06:05 PM, Andres Freund wrote: On 2014-09-26 14:57:12 -0400, Robert Haas wrote: Sure, it'll possibly not be trivial to move them elsewhere. On the other hand, the padding bytes have been unused for 8+ years without somebody laying "claim" on them but "me". I don't think it's a good

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-26 14:57:12 -0400, Robert Haas wrote: > On Fri, Sep 26, 2014 at 12:32 PM, Andres Freund > wrote: > >> Huh? That's just to say that the unused bit space is, in fact, > >> unused. But so what? We've always been very careful about using up > >> things like infomask bits, because there

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Robert Haas
On Fri, Sep 26, 2014 at 12:32 PM, Andres Freund wrote: >> Huh? That's just to say that the unused bit space is, in fact, >> unused. But so what? We've always been very careful about using up >> things like infomask bits, because there are only so many bits >> available, and when they're gone th

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-26 11:02:16 -0400, Robert Haas wrote: > On Fri, Sep 26, 2014 at 10:55 AM, Andres Freund > wrote: > > On 2014-09-26 10:40:37 -0400, Robert Haas wrote: > >> On Fri, Sep 26, 2014 at 10:21 AM, Andres Freund > >> wrote: > >> > As explained above this isn't happening on the level of indivi

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Robert Haas
On Fri, Sep 26, 2014 at 10:55 AM, Andres Freund wrote: > On 2014-09-26 10:40:37 -0400, Robert Haas wrote: >> On Fri, Sep 26, 2014 at 10:21 AM, Andres Freund >> wrote: >> > As explained above this isn't happening on the level of individual AMs. >> >> Well, that's even worse. You want to grab 100

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-26 10:40:37 -0400, Robert Haas wrote: > On Fri, Sep 26, 2014 at 10:21 AM, Andres Freund > wrote: > > As explained above this isn't happening on the level of individual AMs. > > Well, that's even worse. You want to grab 100% of the available > generic bitspace applicable to all record

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Robert Haas
On Fri, Sep 26, 2014 at 10:21 AM, Andres Freund wrote: > As explained above this isn't happening on the level of individual AMs. Well, that's even worse. You want to grab 100% of the available generic bitspace applicable to all record types for purposes specific to logical decoding (and it's sti

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-26 09:53:09 -0400, Robert Haas wrote: > On Fri, Sep 26, 2014 at 5:05 AM, Andres Freund wrote: > >> Let me try to summarize the information requirements for each of these > >> things. For #1, you need to know, after crash recovery, for each > >> standby, the last commit LSN which the cl

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Robert Haas
On Fri, Sep 26, 2014 at 5:05 AM, Andres Freund wrote: >> Let me try to summarize the information requirements for each of these >> things. For #1, you need to know, after crash recovery, for each >> standby, the last commit LSN which the client has confirmed via a >> feedback message. > > I'm not

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-25 22:44:49 -0400, Robert Haas wrote: > Thanks for this write-up. > > On Tue, Sep 23, 2014 at 2:24 PM, Andres Freund wrote: > > 1) The ability Monitor how for replication has progressed in a > >crashsafe manner to allow it to restart at the right point after > >errors/crashes. >

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Petr Jelinek
On 26/09/14 04:44, Robert Haas wrote: On Tue, Sep 23, 2014 at 2:24 PM, Andres Freund wrote: Note that it depends on the replication solution whether these external identifiers need to be coordinated across systems or not. I think it's *good* if we don't propose a solution for that - different

Re: [HACKERS] Replication identifiers, take 3

2014-09-25 Thread Robert Haas
Thanks for this write-up. On Tue, Sep 23, 2014 at 2:24 PM, Andres Freund wrote: > 1) The ability Monitor how for replication has progressed in a >crashsafe manner to allow it to restart at the right point after >errors/crashes. > 2) Efficiently identify the origin of individual changes an

[HACKERS] Replication identifiers, take 3

2014-09-23 Thread Andres Freund
Hi, I've previously started two threads about replication identifiers. Check http://archives.postgresql.org/message-id/20131114172632.GE7522%40alap2.anarazel.de and http://archives.postgresql.org/message-id/20131211153833.GB25227%40awork2.anarazel.de . The've also been discussed in the course of