Re: [HACKERS] The plan for FDW-based sharding

2016-03-11 Thread Bruce Momjian
On Fri, Mar 11, 2016 at 10:19:16AM +0100, Oleg Bartunov wrote: > Our XTM is the yet another example of infrastructure we need to work on > clustering. Should we wait other smart guy starts thinking on distributed > transactions ?  We described in https://wiki.postgresql.org/wiki/DTM our  API, > whi

Re: [HACKERS] The plan for FDW-based sharding

2016-03-11 Thread Oleg Bartunov
On Fri, Mar 11, 2016 at 9:09 AM, Bruce Momjian wrote: > > > > 3. I have tried to encourage others to get involved, with limited > success. I do think the FDW is perhaps the only reasonable way to get > _built-in_ sharding. The external sharding solutions are certainly > viable, but external.

Re: [HACKERS] The plan for FDW-based sharding

2016-03-11 Thread Bruce Momjian
On Fri, Mar 11, 2016 at 04:30:13PM +0800, Craig Ringer wrote: > ... eventually. > > Sometimes the bug reports start. Occasionally you get a "thanks, this looks > interesting/handy". But usually just bug reports or complaints that whatever > you built isn't good enough to meet some random person's

Re: [HACKERS] The plan for FDW-based sharding

2016-03-11 Thread Craig Ringer
On 11 March 2016 at 16:09, Bruce Momjian wrote: > Ideally everything would have a well-defined plan, it is > sometimes hard to do. BDR helped for logical decoding etc - having something concrete really helped shape and guide each part of it as it was (or is/will be, in some cases) migrated fr

Re: [HACKERS] The plan for FDW-based sharding

2016-03-11 Thread Bruce Momjian
I have read the recent comments on this thread with great interest. I am glad people have expressed their concerns, rather than remain silent. Now that the responses have decreased, I can reply. I saw several concerns: 1. My motivation for starting this thread was to decrease interest in extern

Re: [HACKERS] The plan for FDW-based sharding

2016-03-08 Thread Oleg Bartunov
On Tue, Mar 8, 2016 at 6:40 AM, Craig Ringer wrote: > Either that, or bless experimental features/API as an official concept. > I'd quite like that myself - stuff that's in Pg, but documented as "might > change or go away in the next release, experimental feature". As we're > doing more stuff t

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Craig Ringer
On 7 March 2016 at 23:02, Robert Haas wrote: > On Fri, Mar 4, 2016 at 11:17 PM, Craig Ringer > wrote: > > If FDW-based sharding works, I'm happy enough, I have no horse in this > race. > > If it doesn't work I don't much care either. What I'm worried about is > it if > > works like partitioning

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Kevin Grittner
On Mon, Mar 7, 2016 at 6:13 AM, Craig Ringer wrote: > On 5 March 2016 at 23:41, Kevin Grittner wrote: >> The only place you *need* to vary from commit order for correctness >> is when there are overlapping SERIALIZABLE transactions, one >> modifies data and commits, and another reads the old ver

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Robert Haas
On Fri, Mar 4, 2016 at 11:17 PM, Craig Ringer wrote: > If FDW-based sharding works, I'm happy enough, I have no horse in this race. > If it doesn't work I don't much care either. What I'm worried about is it if > works like partitioning using inheritance works - horribly badly, but just > well eno

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Craig Ringer
On 5 March 2016 at 23:41, Kevin Grittner wrote: > > > I'd be really interested in some ideas on how that information might be > > usefully accessed. If we could write info on when to apply commits to the > > xlog in serializable mode that'd be very handy, especially when looking > to > > the futu

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Konstantin Knizhnik
On 03/07/2016 04:28 AM, Robert Haas wrote: On Fri, Mar 4, 2016 at 10:54 PM, Craig Ringer wrote: I've got to say that this is somewhat reminicient of the discussions around in-core pooling, where argument 1 is applied to justify excluding pooling from core/contrib. I don't have a strong positio

Re: [HACKERS] The plan for FDW-based sharding

2016-03-06 Thread Robert Haas
On Fri, Mar 4, 2016 at 10:54 PM, Craig Ringer wrote: > I've got to say that this is somewhat reminicient of the discussions around > in-core pooling, where argument 1 is applied to justify excluding pooling > from core/contrib. > > I don't have a strong position on whether a DTM should be in core

Re: [HACKERS] The plan for FDW-based sharding

2016-03-06 Thread Robert Haas
On Fri, Mar 4, 2016 at 10:23 PM, Craig Ringer wrote: > I can imagine that many such hooks would have little use beyond PPAS, but > I'm somewhat curious as to if any would have wider applications. It's not > unusual for me to be working on something and think "gee, I wish there was a > hook here".

Re: [HACKERS] The plan for FDW-based sharding

2016-03-06 Thread Thom Brown
On 6 Mar 2016 8:27 p.m., "Peter Geoghegan" wrote: > > On Fri, Mar 4, 2016 at 4:41 PM, Robert Haas wrote: > > Yeah, I agree with that. I am utterly mystified by why Bruce keeps > > beating this drum, and am frankly pretty annoyed about it. In the > > first place, he seems to think that he invent

Re: [HACKERS] The plan for FDW-based sharding

2016-03-06 Thread Peter Geoghegan
On Fri, Mar 4, 2016 at 4:41 PM, Robert Haas wrote: > Yeah, I agree with that. I am utterly mystified by why Bruce keeps > beating this drum, and am frankly pretty annoyed about it. In the > first place, he seems to think that he invented the idea of using FDWs > for sharding in PostgreSQL, but I

Re: [HACKERS] The plan for FDW-based sharding

2016-03-05 Thread Kevin Grittner
On Fri, Mar 4, 2016 at 10:10 PM, Craig Ringer wrote: > On 28 February 2016 at 06:38, Kevin Grittner wrote: >> What I sketched out with the "apparent order of execution" >> ordering of the transactions (basically, commit order except >> when one SERIALIZABLE transaction needs to be dragged in fro

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
On 2 March 2016 at 03:02, Bruce Momjian wrote: > On Tue, Mar 1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote: > > Note that I am not saying that other discussed approaches are any > > better, I am saying that we should know approximately what we > > actually want and not just beat FDWs with a ha

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
On 2 March 2016 at 00:03, Robert Haas wrote: > > True. There is an API, though, and having pluggable WAL support seems > desirable too. At the same time, I don't think we know of anyone > maintaining a non-core index AM ... and there are probably good > reasons for that. We end up revising th

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
On 28 February 2016 at 06:38, Kevin Grittner wrote: > > > For logical replay, applying in batches is actually a good thing since it > > allows parallelism. We can remove them all from the target's procarray > all > > at once to avoid intermediate states becoming visible. So that would be > the >

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
On 27 February 2016 at 15:29, Konstantin Knizhnik wrote: > Two reasons: > 1. There is no ideal implementation of DTM which will fit all possible > needs and be efficient for all clusters. > 2. Even if such implementation exists, still the right way of it > integration is Postgres should use kin

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
On 27 February 2016 at 11:54, Robert Haas wrote: > I could submit a patch adding > hooks to core to enable all of the things (or even just some of the > things) that EnterpriseDB has changed in Advanced Server, and that > patch would be rejected so fast it would make your head spin, because > o

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Robert Haas
On Fri, Mar 4, 2016 at 8:27 PM, Joshua D. Drake wrote: > This does not sound like Bruce at all. Bruce is a lot of things, stubborn, > sometimes temperamental, a lot of times like you... a hot head but he does > not take credit for other people's work in my experience. On the whole, Bruce is a muc

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Joshua D. Drake
On 03/04/2016 04:41 PM, Robert Haas wrote: As far as I understand it, Bruce came in near the end of that conversation and now wants to claim credit for something that doesn't really exist yet and, to the extent that it does exist, wasn't even his idea. Robert, This does not sound like Bruce at

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Robert Haas
On Tue, Mar 1, 2016 at 12:07 PM, Konstantin Knizhnik wrote: > In the article them used anotion "wait": > > if T.SnapshotTime>GetClockTime() > then wait until T.SnapshotTime > Originally we really do sleep here, but then we think that instead of > sleeping we can just adjust local time. > Sorry, I

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Robert Haas
On Wed, Mar 2, 2016 at 1:53 PM, Josh berkus wrote: > One of the things which causes bad reactions and arguments, Bruce, is that a > lot of your posts and presentations detailing plans for the FDW approach > carry the subtext that all four of the other approaches are dead ends and > not worth consi

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Oleg Bartunov
On Mar 3, 2016 4:47 AM, "Michael Paquier" wrote: > > On Wed, Mar 2, 2016 at 6:54 PM, Alexander Korotkov > wrote: > > If FDWs existed then Postgres XC/XL were being developed then I believe they > > would try to build full-featured prototype of FDW based sharding. If this > > prototype succeed the

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Tatsuo Ishii
> On Wed, Mar 2, 2016 at 6:54 PM, Alexander Korotkov > wrote: >> If FDWs existed then Postgres XC/XL were being developed then I believe they >> would try to build full-featured prototype of FDW based sharding. If this >> prototype succeed then we could make a full roadmap. > > Speaking here with

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Michael Paquier
On Wed, Mar 2, 2016 at 6:54 PM, Alexander Korotkov wrote: > If FDWs existed then Postgres XC/XL were being developed then I believe they > would try to build full-featured prototype of FDW based sharding. If this > prototype succeed then we could make a full roadmap. Speaking here with my XC hat,

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Alexander Korotkov
On Wed, Mar 2, 2016 at 9:53 PM, Josh berkus wrote: > On 02/24/2016 01:22 AM, Konstantin Knizhnik wrote: > >> Sorry, but based on this plan it is possible to make a conclusion that >> there are only two possible cluster solutions for Postgres: >> XC/XL and FDW-based. From my point of view there a

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Josh berkus
On 02/24/2016 01:22 AM, Konstantin Knizhnik wrote: Sorry, but based on this plan it is possible to make a conclusion that there are only two possible cluster solutions for Postgres: XC/XL and FDW-based. From my point of view there are much more possible alternatives. Definitely. Currently we

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Konstantin Knizhnik
On 01.03.2016 22:02, Bruce Momjian wrote: On Tue, Mar 1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote: Note that I am not saying that other discussed approaches are any better, I am saying that we should know approximately what we actually want and not just beat FDWs with a hammer and hope sh

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Alexander Korotkov
On Tue, Mar 1, 2016 at 10:11 PM, Bruce Momjian wrote: > On Tue, Mar 1, 2016 at 02:02:44PM -0500, Bruce wrote: > > On Tue, Mar 1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote: > > > Note that I am not saying that other discussed approaches are any > > > better, I am saying that we should know ap

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Alexander Korotkov
On Tue, Mar 1, 2016 at 7:03 PM, Robert Haas wrote: > On Tue, Mar 1, 2016 at 10:37 AM, Bruce Momjian wrote: > > On Tue, Mar 1, 2016 at 10:19:45AM -0500, Robert Haas wrote: > >> > Two reasons: > >> > 1. There is no ideal implementation of DTM which will fit all > possible needs > >> > and be eff

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Oleg Bartunov
On Wed, Mar 2, 2016 at 4:36 AM, Tomas Vondra wrote: Hi, > > On 03/01/2016 08:02 PM, Bruce Momjian wrote: > >> On Tue, Mar 1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote: >> >>> Note that I am not saying that other discussed approaches are any >>> better, I am saying that we should know approxim

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Oleg Bartunov
On Tue, Mar 1, 2016 at 7:03 PM, Robert Haas wrote: > On Tue, Mar 1, 2016 at 10:37 AM, Bruce Momjian wrote: > > On Tue, Mar 1, 2016 at 10:19:45AM -0500, Robert Haas wrote: > >> > Two reasons: > >> > 1. There is no ideal implementation of DTM which will fit all > possible needs > >> > and be eff

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Tomas Vondra
Hi, On 03/01/2016 08:02 PM, Bruce Momjian wrote: On Tue, Mar 1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote: Note that I am not saying that other discussed approaches are any better, I am saying that we should know approximately what we actually want and not just beat FDWs with a hammer and h

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
On 03/01/2016 09:19 PM, Petr Jelinek wrote: Since this thread heavily discusses the XTM, I have question about the XTM as proposed because one thing is very unclear to me - what happens when user changes the XTM plugin on the server? I didn't see any xid handover API which makes me wonder if ch

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Bruce Momjian
On Tue, Mar 1, 2016 at 02:02:44PM -0500, Bruce wrote: > On Tue, Mar 1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote: > > Note that I am not saying that other discussed approaches are any > > better, I am saying that we should know approximately what we > > actually want and not just beat FDWs wit

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Bruce Momjian
On Tue, Mar 1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote: > Note that I am not saying that other discussed approaches are any > better, I am saying that we should know approximately what we > actually want and not just beat FDWs with a hammer and hope sharding > will eventually emerge and call

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Petr Jelinek
On 27/02/16 04:54, Robert Haas wrote: On Fri, Feb 26, 2016 at 10:56 PM, Konstantin Knizhnik wrote: We do not have formal prove that proposed XTM is "general enough" to handle all possible transaction manager implementations. But there are two general ways of dealing with isolation: snapshot bas

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Petr Jelinek
On 01/03/16 18:18, Konstantin Knizhnik wrote: On 01.03.2016 19:03, Robert Haas wrote: On Tue, Mar 1, 2016 at 10:37 AM, Bruce Momjian wrote: On Tue, Mar 1, 2016 at 10:19:45AM -0500, Robert Haas wrote: Two reasons: 1. There is no ideal implementation of DTM which will fit all possible needs a

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
On 01.03.2016 19:03, Robert Haas wrote: On Tue, Mar 1, 2016 at 10:37 AM, Bruce Momjian wrote: On Tue, Mar 1, 2016 at 10:19:45AM -0500, Robert Haas wrote: Two reasons: 1. There is no ideal implementation of DTM which will fit all possible needs and be efficient for all clusters. Hmm, what

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
Thank you very much for you comments. On 01.03.2016 18:19, Robert Haas wrote: On Sat, Feb 27, 2016 at 2:29 AM, Konstantin Knizhnik wrote: How do you prevent clock skew from causing serialization anomalies? If node receives message from "feature" it just needs to wait until this future arrive.

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Robert Haas
On Tue, Mar 1, 2016 at 10:37 AM, Bruce Momjian wrote: > On Tue, Mar 1, 2016 at 10:19:45AM -0500, Robert Haas wrote: >> > Two reasons: >> > 1. There is no ideal implementation of DTM which will fit all possible >> > needs >> > and be efficient for all clusters. >> >> Hmm, what is the reasoning b

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Bruce Momjian
On Tue, Mar 1, 2016 at 10:19:45AM -0500, Robert Haas wrote: > > Two reasons: > > 1. There is no ideal implementation of DTM which will fit all possible needs > > and be efficient for all clusters. > > Hmm, what is the reasoning behind that statement? I mean, it is > certainly true that there ar

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Robert Haas
On Sat, Feb 27, 2016 at 2:29 AM, Konstantin Knizhnik wrote: >> How do you prevent clock skew from causing serialization anomalies? > > If node receives message from "feature" it just needs to wait until this > future arrive. > Practically we just "adjust" system time in this case, moving it forwar

Re: [HACKERS] The plan for FDW-based sharding

2016-02-28 Thread Simon Riggs
On 27 February 2016 at 22:38, Kevin Grittner wrote: > That could be part of a solution. What I sketched out with the > "apparent order of execution" ordering of the transactions > (basically, commit order except when one SERIALIZABLE transaction > needs to be dragged in front of another due to

Re: [HACKERS] The plan for FDW-based sharding

2016-02-28 Thread Konstantin Knizhnik
On 02/27/2016 11:38 PM, Kevin Grittner wrote: Is this an implementation of some particular formal technique? If so, do you have a reference to a paper on it? I get the sense that there has been a lot written about distributed transactions, and that it would be a mistake to ignore it, but I hav

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Kevin Grittner
On Sat, Feb 27, 2016 at 3:57 PM, Simon Riggs wrote: > On 27 February 2016 at 17:54, Kevin Grittner wrote: >> >> On a single database SSI can see whether a read has >> caused such a problem. If you replicate the transactions to >> somewhere else and read them SSI cannot tell whether there is an >

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Simon Riggs
On 27 February 2016 at 17:54, Kevin Grittner wrote: > On a single database SSI can see whether a read has > caused such a problem. If you replicate the transactions to > somewhere else and read them SSI cannot tell whether there is an > anomaly OK, I thought you were saying something else. Wha

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Kevin Grittner
On Sat, Feb 27, 2016 at 1:14 PM, Konstantin Knizhnik wrote: > We do not try to preserve transaction commit order at all nodes. > But in principle it can be implemented using XTM API: it allows to redefine > function which actually sets transaction status. pg_dtm performs 2PC here. > And in princ

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Konstantin Knizhnik
Neither pg_dtm, neither pg_tsdtm supports serializable isolation level. We implemented distributed snapshot isolation - repeatable-read isolation level. We also do not support read-committed isolation level now. We do not try to preserve transaction commit order at all nodes. But in principle it

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Kevin Grittner
On Fri, Feb 26, 2016 at 5:37 PM, Simon Riggs wrote: > On 26 February 2016 at 22:48, Kevin Grittner wrote: >> if we want logical >> replication to be free of serialization anomalies for those using >> serializable transactions, we need to support applying transactions >> in an order which may not

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Álvaro Hernández Tortosa
On 27/02/16 09:19, Konstantin Knizhnik wrote: On 02/27/2016 06:54 AM, Robert Haas wrote: [...] So maybe the goal for the GTM isn't to provide true serializability across the cluster but some lesser degree of transaction isolation. But then exactly which serialization anomalies are we tryi

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Konstantin Knizhnik
On 02/27/2016 06:54 AM, Robert Haas wrote: On Fri, Feb 26, 2016 at 10:56 PM, Konstantin Knizhnik wrote: We do not have formal prove that proposed XTM is "general enough" to handle all possible transaction manager implementations. But there are two general ways of dealing with isolation: snapsho

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
On 02/27/2016 06:57 AM, Robert Haas wrote: On Sat, Feb 27, 2016 at 1:49 AM, Konstantin Knizhnik wrote: pg_tsdtm is based on another approach: it is using system time as CSN and doesn't require arbiter. In theory there is no limit for scalability. But differences in system time and necessity to

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Robert Haas
On Sat, Feb 27, 2016 at 1:49 AM, Konstantin Knizhnik wrote: > pg_tsdtm is based on another approach: it is using system time as CSN and > doesn't require arbiter. In theory there is no limit for scalability. But > differences in system time and necessity to use more rounds of communication > have

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Robert Haas
On Fri, Feb 26, 2016 at 10:56 PM, Konstantin Knizhnik wrote: > We do not have formal prove that proposed XTM is "general enough" to handle > all possible transaction manager implementations. > But there are two general ways of dealing with isolation: snapshot based and > CSN based. I don't belie

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Simon Riggs
On 26 February 2016 at 22:48, Kevin Grittner wrote: > On Fri, Feb 26, 2016 at 2:19 PM, Konstantin Knizhnik > wrote: > > > pg_tsdtm is based on another approach: it is using system time > > as CSN > > Which brings up an interesting point, if we want logical > replication to be free of serializat

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Kevin Grittner
On Fri, Feb 26, 2016 at 2:19 PM, Konstantin Knizhnik wrote: > pg_tsdtm is based on another approach: it is using system time > as CSN Which brings up an interesting point, if we want logical replication to be free of serialization anomalies for those using serializable transactions, we need to

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
On 02/26/2016 09:30 PM, Alvaro Herrera wrote: Konstantin Knizhnik wrote: Yes, it is certainly possible to develop cluster by cloning PostgreSQL. But it cause big problems both for developers, which have to permanently synchronize their branch with master, and, what is more important, for custom

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Bruce Momjian
On Fri, Feb 26, 2016 at 03:30:29PM -0300, Alvaro Herrera wrote: > That's not the point, though. I don't think a Postgres clone with a GTM > solves any particular problem that's not already solved by the existing > forks. However, if you have a clone at home and you make a GTM work on > it, then y

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Alvaro Herrera
Konstantin Knizhnik wrote: > Yes, it is certainly possible to develop cluster by cloning PostgreSQL. > But it cause big problems both for developers, which have to permanently > synchronize their branch with master, > and, what is more important, for customers, which can not use standard > version

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
We do not have formal prove that proposed XTM is "general enough" to handle all possible transaction manager implementations. But there are two general ways of dealing with isolation: snapshot based and CSN based. pg_dtm and pg_tsdtm prove that both of them can be implemented using XTM. If you

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Robert Haas
On Fri, Feb 26, 2016 at 10:00 PM, Joshua D. Drake wrote: > Robert, this is all a game. It is a game of who wins the intellectual prize > to whatever problem. Who gets the market or mind share and who gets to > pretend they win the Oscar for coolest design. JD, I don't have a horse in this race.

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Joshua D. Drake
On 02/26/2016 08:06 AM, Robert Haas wrote: On Fri, Feb 26, 2016 at 7:21 PM, Oleg Bartunov wrote: Right now tm is hardcoded and it's doesn't matter "if other people might need" at all. We at least provide developers ("other people") ability to work on their implementations and the patch is s

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Robert Haas
On Fri, Feb 26, 2016 at 7:21 PM, Oleg Bartunov wrote: > Right now tm is hardcoded and it's doesn't matter "if other people might > need" at all. We at least provide developers ("other people") ability to > work on their implementations and the patch is safe and doesn't sacrifices > anything in

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Oleg Bartunov
On Fri, Feb 26, 2016 at 3:50 PM, Robert Haas wrote: > On Wed, Feb 24, 2016 at 3:05 PM, Oleg Bartunov > wrote: > > I already several times pointed, that we need XTM to be able to continue > > development in different directions, since there is no clear winner. > > Moreover, I think there is no fi

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Robert Haas
On Wed, Feb 24, 2016 at 3:05 PM, Oleg Bartunov wrote: > I already several times pointed, that we need XTM to be able to continue > development in different directions, since there is no clear winner. > Moreover, I think there is no fits-all solution and while I agree we need > one built-in in the

Re: [HACKERS] The plan for FDW-based sharding

2016-02-25 Thread Bruce Momjian
On Thu, Feb 25, 2016 at 01:53:12PM +0900, Michael Paquier wrote: > > Well, as far as I know XC doesn't support data redistribution between > > nodes and I saw good benchmarks of that, as well as XL. > > XC does support that in 1.2 with a very basic approach (coded that > years ago), though it take

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Michael Paquier
On Wed, Feb 24, 2016 at 11:34 PM, Bruce Momjian wrote: > On Wed, Feb 24, 2016 at 12:17:28PM +0300, Alexander Korotkov wrote: >> Hi, Bruce! >> >> The important point for me is to distinguish different kind of plans: >> implementation plan and research plan. >> If we're talking about implementation

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Bruce Momjian
On Wed, Feb 24, 2016 at 01:02:21PM -0300, Alvaro Herrera wrote: > Bruce Momjian wrote: > > On Wed, Feb 24, 2016 at 01:08:29AM +, Simon Riggs wrote: > > > > It's never been our policy to try to include major projects in single code > > > drops. Any move of XL/XC code into PostgreSQL core would

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Alvaro Herrera
Bruce Momjian wrote: > On Wed, Feb 24, 2016 at 01:08:29AM +, Simon Riggs wrote: > > It's never been our policy to try to include major projects in single code > > drops. Any move of XL/XC code into PostgreSQL core would need to be done > > piece > > by piece across many releases. XL is defini

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Bruce Momjian
On Wed, Feb 24, 2016 at 09:34:37AM -0500, Bruce Momjian wrote: > > I have nothing against particular FDW advances. However, it's unclear for me > > that FDW should be the only sharding approach. > > It's unproven that FDW can do work that Postgres XC/XL does. With FDW we can > > have some low-hangi

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Bruce Momjian
On Wed, Feb 24, 2016 at 12:22:20PM +0300, Konstantin Knizhnik wrote: > Sorry, but based on this plan it is possible to make a conclusion > that there are only two possible cluster solutions for Postgres: > XC/XL and FDW-based. From my point of view there are much more > possible alternatives. > O

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Bruce Momjian
On Wed, Feb 24, 2016 at 12:35:15PM +0300, Oleg Bartunov wrote: > I have nothing against particular FDW advances. However, it's unclear for > me that FDW should be the only sharding approach. > It's unproven that FDW can do work that Postgres XC/XL does. With FDW we > can have some l

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Bruce Momjian
On Wed, Feb 24, 2016 at 12:17:28PM +0300, Alexander Korotkov wrote: > Hi, Bruce! > > The important point for me is to distinguish different kind of plans: > implementation plan and research plan. > If we're talking about implementation plan then it should be proven that > proposed approach works i

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Oleg Bartunov
On Wed, Feb 24, 2016 at 12:17 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > Hi, Bruce! > > The important point for me is to distinguish different kind of plans: > implementation plan and research plan. > If we're talking about implementation plan then it should be proven that > prop

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Konstantin Knizhnik
Sorry, but based on this plan it is possible to make a conclusion that there are only two possible cluster solutions for Postgres: XC/XL and FDW-based. From my point of view there are much more possible alternatives. Our main idea with XTM (eXtensible Transaction Manager API) was to make it po

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Alexander Korotkov
Hi, Bruce! The important point for me is to distinguish different kind of plans: implementation plan and research plan. If we're talking about implementation plan then it should be proven that proposed approach works in this case. I.e research should be already done. If we're talking about researc

Re: [HACKERS] The plan for FDW-based sharding

2016-02-23 Thread Bruce Momjian
On Wed, Feb 24, 2016 at 01:08:29AM +, Simon Riggs wrote: > On 23 February 2016 at 16:43, Bruce Momjian wrote: > > There was discussion at the FOSDEM/PGDay Developer Meeting > (https://wiki.postgresql.org/wiki/FOSDEM/PGDay_2016_Developer_Meeting) > about sharding so I wanted to out

Re: [HACKERS] The plan for FDW-based sharding

2016-02-23 Thread Simon Riggs
On 23 February 2016 at 16:43, Bruce Momjian wrote: > There was discussion at the FOSDEM/PGDay Developer Meeting > (https://wiki.postgresql.org/wiki/FOSDEM/PGDay_2016_Developer_Meeting) > about sharding so I wanted to outline where I think we are going with > sharding and FDWs. > I think we need

Re: [HACKERS] The plan for FDW-based sharding

2016-02-23 Thread Bruce Momjian
On Tue, Feb 23, 2016 at 09:54:46AM -0700, David G. Johnston wrote: > On Tue, Feb 23, 2016 at 9:43 AM, Bruce Momjian wrote: > > 4. Cross-node read-write queries: > > This will require a global snapshot manager and global snapshot manager. > > > Probably meant "global transaction manager

Re: [HACKERS] The plan for FDW-based sharding

2016-02-23 Thread David G. Johnston
On Tue, Feb 23, 2016 at 9:43 AM, Bruce Momjian wrote: > 4. Cross-node read-write queries: > > This will require a global snapshot manager and global snapshot manager. > Probably meant "global transaction manager" ​David J.​

[HACKERS] The plan for FDW-based sharding

2016-02-23 Thread Bruce Momjian
There was discussion at the FOSDEM/PGDay Developer Meeting (https://wiki.postgresql.org/wiki/FOSDEM/PGDay_2016_Developer_Meeting) about sharding so I wanted to outline where I think we are going with sharding and FDWs. First, let me point out that, unlike pg_upgrade and the Windows port, which eit