Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-20 Thread Andres Freund
Hi Robert, Hi All! On Wednesday, June 20, 2012 03:08:48 AM Robert Haas wrote: > On Tue, Jun 19, 2012 at 2:23 PM, Andres Freund wrote: > >> Well, the words are fuzzy, but I would define logical replication to > >> be something which is independent of the binary format in which stuff > >> gets sto

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 2:23 PM, Andres Freund wrote: >> Well, the words are fuzzy, but I would define logical replication to >> be something which is independent of the binary format in which stuff >> gets stored on disk.  If it's not independent of the disk format, then >> you can't do heterogen

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Kevin Grittner
Andres Freund wrote: > The problem is just that to support basically arbitrary decoding > requirements you need to provide at least those pieces of > information in a transactionally consistent manner: > * the data > * table names > * column names > * type information > * replication configurati

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Andres Freund
Hi, The most important part, even for people not following my discussion with Robert is at the bottom where the possible wal decoding strategies are laid out. On Tuesday, June 19, 2012 03:20:58 AM Robert Haas wrote: > On Sat, Jun 16, 2012 at 7:43 AM, Andres Freund wrote: > >> > Hm. Yes, you c

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-18 Thread Robert Haas
On Sat, Jun 16, 2012 at 7:43 AM, Andres Freund wrote: >> > Hm. Yes, you could do that. But I have to say I don't really see a point. >> > Maybe the fact that I do envision multimaster systems at some point is >> > clouding my judgement though as its far less easy in that case. >> Why?  I don't thi

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-18 Thread Robert Haas
On Sat, Jun 16, 2012 at 3:03 PM, Steve Singer wrote: > I feel that in-core support to capture changes and turn them into change > records that can be replayed on other databases, without relying on triggers > and log tables, would be good to have. > > I think we want some flexible enough that peop

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-16 Thread Steve Singer
On 12-06-15 04:03 PM, Robert Haas wrote: On Thu, Jun 14, 2012 at 4:13 PM, Andres Freund wrote: I don't plan to throw in loads of conflict resolution smarts. The aim is to get to the place where all the infrastructure is there so that a MM solution can be built by basically plugging in a conflic

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-16 Thread Andres Freund
Hi Robert, On Friday, June 15, 2012 10:03:38 PM Robert Haas wrote: > On Thu, Jun 14, 2012 at 4:13 PM, Andres Freund wrote: > > I don't plan to throw in loads of conflict resolution smarts. The aim is > > to get to the place where all the infrastructure is there so that a MM > > solution can be b

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-15 Thread Kevin Grittner
Robert Haas wrote: > So maybe instead of trying to cobble together a set of catalog > contents that we can use for decoding any tuple whatsoever, we > should instead divide the world into well-behaved types and > poorly-behaved types. Well-behaved types are those that can be > interpreted without

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-15 Thread Robert Haas
On Thu, Jun 14, 2012 at 4:13 PM, Andres Freund wrote: > I don't plan to throw in loads of conflict resolution smarts. The aim is to > get > to the place where all the infrastructure is there so that a MM solution can > be built by basically plugging in a conflict resolution mechanism. Maybe > pro

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-14 Thread Andres Freund
Hi Robert, Thanks for your answer. On Thursday, June 14, 2012 06:17:26 PM Robert Haas wrote: > On Wed, Jun 13, 2012 at 7:27 AM, Andres Freund wrote: > > === Design goals for logical replication === : > > - in core > > - fast > > - async > > - robust > > - multi-master > > - modular > > - as uni

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-14 Thread Robert Haas
On Wed, Jun 13, 2012 at 7:27 AM, Andres Freund wrote: > === Design goals for logical replication === : > - in core > - fast > - async > - robust > - multi-master > - modular > - as unintrusive as possible implementation wise > - basis for other technologies (sharding, replication into other DBMSs,

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
Hi, The patch as of yet doesn't contain how you actually can use the prototype... Obviously at this point its not very convenient: I have two config files: Node 1: port = 5501 wal_level = logical max_wal_senders = 10 wal_keep_segments = 200 multimaster_conninfo = 'port=5502 host=/tmp' multimaste

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
On Wednesday, June 13, 2012 07:11:40 PM Steve Singer wrote: > On 12-06-13 07:27 AM, Andres Freund wrote: > > Its also available in the 'cabal-rebasing' branch on > > git.postgresql.org/users/andresfreund/postgres.git . That branch will > > modify history though. > > That branch has a merge error i

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
On Wednesday, June 13, 2012 07:11:40 PM Steve Singer wrote: > On 12-06-13 07:27 AM, Andres Freund wrote: > > Its also available in the 'cabal-rebasing' branch on > > git.postgresql.org/users/andresfreund/postgres.git . That branch will > > modify history though. > > That branch has a merge error i

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Steve Singer
On 12-06-13 07:27 AM, Andres Freund wrote: Its also available in the 'cabal-rebasing' branch on git.postgresql.org/users/andresfreund/postgres.git . That branch will modify history though. That branch has a merge error in f685a11ce43b9694cbe61ffa42e396c9fbc32b05 gcc -O2 -Wall -Wmissing-proto

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
On Wednesday, June 13, 2012 05:53:31 PM Robert Haas wrote: > On Wed, Jun 13, 2012 at 7:27 AM, Andres Freund wrote: > > Unless somebody objects I will add most of the individual marked as RFC > > to the current commitfest. I hope that with comments stemming from that > > round we can get several o

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Robert Haas
On Wed, Jun 13, 2012 at 7:27 AM, Andres Freund wrote: > Unless somebody objects I will add most of the individual marked as RFC to the > current commitfest. I hope that with comments stemming from that round we can > get several of the patches into the first or second commitfest. As soon as the >

[HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
Hi everyone, This mail contains the highlevel design description of how our prototype of in-core logical replication works. The individual patches will be posted as replies to this email. I obviously welcome all sorts of productive comments to both the individual patches and the architecture. Unl