Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2

2012-10-25 Thread Alvaro Herrera
> Comments about the approach or even the general direction of the > implementation? Questions? This patch series has gotten serious amount of discussion and useful feedback; even some parts of it have been committed. I imagine lots more feedback, discussion and spawning of new ideas will take p

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-18 Thread Peter Geoghegan
On 18 October 2012 16:18, Christopher Browne wrote: > A "shim" adds complexity, but retains the "upgrade across versions" > use case, and reduces the need to keep supporting elder versions of > Slony. Right. Upgrading across major versions is likely to continue to remain a very important use-case

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-18 Thread Christopher Browne
On Thu, Oct 18, 2012 at 9:49 AM, Peter Geoghegan wrote: > On 16 October 2012 15:26, Jan Wieck wrote: >> This means that the transition time from the existing, trigger based >> approach to the new WAL based mechanism will see both technologies in >> parallel, which is no small thing to support. >

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-18 Thread Peter Geoghegan
On 16 October 2012 15:26, Jan Wieck wrote: > This means that the transition time from the existing, trigger based > approach to the new WAL based mechanism will see both technologies in > parallel, which is no small thing to support. So, you're talking about a shim between the two in order to use

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-16 Thread Jan Wieck
On 10/15/2012 3:25 PM, Andres Freund wrote: On Monday, October 15, 2012 09:18:57 PM Peter Geoghegan wrote: On 15 October 2012 19:19, Bruce Momjian wrote: > I think Robert is right that if Slony can't use the API, it is unlikely > any other replication system could use it. I don't accept that.

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-16 Thread Jan Wieck
On 10/15/2012 4:43 PM, Simon Riggs wrote: Jan spoke at length at PgCon, for all to hear, that what we are building is a much better way than the trigger logging approach Slony uses. I don't take that as carte blanche for approval of everything being done, but its going in the right direction with

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Steve Singer
On 12-10-15 04:51 PM, Andres Freund wrote: Well, as a crosscheck, could you list your requirements? Do you need anything more than outputting data in a format compatible to whats stored in sl_log_*? You wouldn't have sl_actionseq, everything else should be there (Well, you would need to do look

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Tuesday, October 16, 2012 12:13:14 AM Christopher Browne wrote: > On Mon, Oct 15, 2012 at 4:51 PM, Andres Freund wrote: > > On Monday, October 15, 2012 10:08:28 PM Christopher Browne wrote: > >> On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan > > > > wrote: > >> > On 15 October 2012 19:19,

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Christopher Browne
On Mon, Oct 15, 2012 at 4:51 PM, Andres Freund wrote: > On Monday, October 15, 2012 10:08:28 PM Christopher Browne wrote: >> On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan > wrote: >> > On 15 October 2012 19:19, Bruce Momjian wrote: >> >> I think Robert is right that if Slony can't use the API

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 10:08:28 PM Christopher Browne wrote: > On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan wrote: > > On 15 October 2012 19:19, Bruce Momjian wrote: > >> I think Robert is right that if Slony can't use the API, it is unlikely > >> any other replication system could use

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Simon Riggs
On 15 October 2012 21:03, Tom Lane wrote: > Robert Haas writes: >> On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan >> wrote: >>> On 15 October 2012 19:19, Bruce Momjian wrote: I think Robert is right that if Slony can't use the API, it is unlikely any other replication system could

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 10:03:40 PM Tom Lane wrote: > Robert Haas writes: > > On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan wrote: > >> On 15 October 2012 19:19, Bruce Momjian wrote: > >>> I think Robert is right that if Slony can't use the API, it is unlikely > >>> any other replication

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Christopher Browne
On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan wrote: > On 15 October 2012 19:19, Bruce Momjian wrote: >> I think Robert is right that if Slony can't use the API, it is unlikely >> any other replication system could use it. > > I don't accept that. Clearly there is a circular dependency, and >

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Tom Lane
Robert Haas writes: > On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan > wrote: >> On 15 October 2012 19:19, Bruce Momjian wrote: >>> I think Robert is right that if Slony can't use the API, it is unlikely >>> any other replication system could use it. >> I don't accept that. Clearly there is

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 09:18:57 PM Peter Geoghegan wrote: > On 15 October 2012 19:19, Bruce Momjian wrote: > > I think Robert is right that if Slony can't use the API, it is unlikely > > any other replication system could use it. > > I don't accept that. Clearly there is a circular dependenc

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Robert Haas
On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan wrote: > On 15 October 2012 19:19, Bruce Momjian wrote: >> I think Robert is right that if Slony can't use the API, it is unlikely >> any other replication system could use it. > > I don't accept that. Clearly there is a circular dependency, and >

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Peter Geoghegan
On 15 October 2012 19:19, Bruce Momjian wrote: > I think Robert is right that if Slony can't use the API, it is unlikely > any other replication system could use it. I don't accept that. Clearly there is a circular dependency, and someone has to go first - why should the Slony guys invest in adop

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Hannu Krosing
On 10/15/2012 04:54 AM, Robert Haas wrote: PS. I'd love to see a basic Slony plugin for this, for example, to see how >much extra code on top of the posted patches you need to write in a plugin >like that to make it functional. I'm worried that it's a lot.. I agree. I would go so far as to say

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Hannu Krosing
On 10/15/2012 08:44 PM, Andres Freund wrote: On Monday, October 15, 2012 08:38:07 PM Hannu Krosing wrote: On 10/11/2012 01:42 PM, Andres Freund wrote: On Thursday, October 11, 2012 09:15:47 AM Heikki Linnakangas wrote: ... If the only meaningful advantage is reducing the amount of WAL written,

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Hannu Krosing
On 10/15/2012 04:54 AM, Robert Haas wrote: On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas wrote: IMHO that's a good thing, and I'd hope this new logical replication to live outside core as well, as much as possible. But whether or not something is in core is just a political decision, not

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 08:38:07 PM Hannu Krosing wrote: > On 10/11/2012 01:42 PM, Andres Freund wrote: > > On Thursday, October 11, 2012 09:15:47 AM Heikki Linnakangas wrote: > > ... > > If the only meaningful advantage is reducing the amount of WAL written, > > I can't help thinking that we s

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Hannu Krosing
On 10/11/2012 01:42 PM, Andres Freund wrote: On Thursday, October 11, 2012 09:15:47 AM Heikki Linnakangas wrote: ... If the only meaningful advantage is reducing the amount of WAL written, I can't help thinking that we should just try to address that in the existing solutions, even if it seems "e

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 08:19:54 PM Bruce Momjian wrote: > On Mon, Oct 15, 2012 at 09:57:19AM +0200, Andres Freund wrote: > > On Monday, October 15, 2012 04:54:20 AM Robert Haas wrote: > > > On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas > > > > > > wrote: > > > > IMHO that's a good thin

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Bruce Momjian
On Mon, Oct 15, 2012 at 09:57:19AM +0200, Andres Freund wrote: > On Monday, October 15, 2012 04:54:20 AM Robert Haas wrote: > > On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas > > > > wrote: > > > IMHO that's a good thing, and I'd hope this new logical replication to > > > live outside core a

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 04:54:20 AM Robert Haas wrote: > On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas > > wrote: > > IMHO that's a good thing, and I'd hope this new logical replication to > > live outside core as well, as much as possible. But whether or not > > something is in core is

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-14 Thread Robert Haas
On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas wrote: > IMHO that's a good thing, and I'd hope this new logical replication to live > outside core as well, as much as possible. But whether or not something is > in core is just a political decision, not a reason to implement something > new. >

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-11 Thread Andres Freund
On Thursday, October 11, 2012 09:15:47 AM Heikki Linnakangas wrote: > On 22.09.2012 20:00, Andres Freund wrote: > > [[basic-schema]] > > .Architecture Schema > > ["ditaa"] > > - > > - > > > > Traditional Stuff > >

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-11 Thread Heikki Linnakangas
On 22.09.2012 20:00, Andres Freund wrote: [[basic-schema]] .Architecture Schema ["ditaa"] -- Traditional Stuff +-+-+-+-++ | Backend | Backend | Backend | Autovac | ...|

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-09-22 Thread m...@rpzdesign.com
Andres, nice job on the writeup. I think one aspect you are missing is that there must be some way for the multi-masters to re-stabilize their data sets and quantify any data loss. You cannot do this without some replication intelligence in each row of each table so that no matter how disastr

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents

2012-09-22 Thread Andres Freund
Hi all, Attached is the .txt and .pdf (both are imo readable and contain the same content) with design documentation about the proposed feature. Christan Kruse, Marko Tiikkaja and Hannu Krosing read the document and told me about my most egregious mistakes. Thanks! I would appreciate some feed

[HACKERS] [RFC][PATCH] wal decoding, attempt #2

2012-09-14 Thread Andres Freund
Hi, It took me far longer than I planned, its not finished, but time is running out. I would like some feedback that I am not going astray at this point... *I* think the general approach is sound and a good way forward that provides the basic infrastructure for many (all?) of the scenarios we t