On Mon, Sep 7, 2015 at 1:33 PM, Ahsan Hadi wrote:
> I
>
> On Monday, September 7, 2015, Ashutosh Bapat
> wrote:
>>
>>
>>
>> On Sat, Sep 5, 2015 at 4:22 AM, Ozgun Erdogan wrote:
>>>
>>> Hey Robert,
>>>
Now the question is, where should the code that does all of this live?
postgres_fdw?
On 2015-09-08 19:52, Robert Haas wrote:
On Fri, Sep 4, 2015 at 6:52 PM, Ozgun Erdogan wrote:
For Citus, a second part of the question is as FDW writers. We implemented
cstore_fdw, json_fdw, and mongo_fdw, and these wrappers don't benefit from
even the simple join pushdown that doesn't require
On Fri, Sep 4, 2015 at 6:52 PM, Ozgun Erdogan wrote:
> Distributed shuffles (Map/Reduce) are hard. When we looked at using FDWs for
> pg_shard, we thought that Map/Reduce would require a comprehensive revamp of
> the APIs.
Well, so you've said. But what kind of API do you want to see?
Taking con
On Thu, Sep 3, 2015 at 7:03 AM, Josh Berkus wrote:
> On 09/02/2015 11:41 AM, Robert Haas wrote:
> > 4. Therefore, I think that we should instead use logical replication,
> > which might be either synchronous or asynchronous. When you modify
> > one copy of the data, that change will then be repl
I
On Monday, September 7, 2015, Ashutosh Bapat <
ashutosh.ba...@enterprisedb.com> wrote:
>
>
> On Sat, Sep 5, 2015 at 4:22 AM, Ozgun Erdogan > wrote:
>
>> Hey Robert,
>>
>> Now the question is, where should the code that does all of this live?
>>> postgres_fdw? Some new, sharding-specific FDW?
On Sat, Sep 5, 2015 at 4:22 AM, Ozgun Erdogan wrote:
> Hey Robert,
>
> Now the question is, where should the code that does all of this live?
>> postgres_fdw? Some new, sharding-specific FDW? In core? I don't
>> know for sure, but what I do know is that we could make a lot of
>> progress over
Hey Robert,
Now the question is, where should the code that does all of this live?
> postgres_fdw? Some new, sharding-specific FDW? In core? I don't
> know for sure, but what I do know is that we could make a lot of
> progress over where we are today by just improving postgres_fdw, and I
> don
On 09/03/2015 03:57 AM, Bruce Momjian wrote:
>> >
>> > Yes, the logical replication has similar syncrep properties as the
>> > binary one (feedback works same way).
Oh? What does UDR/BDR currently support for sync?
> Yes, I assumed that. Logical replication uses WAL, so if you are
> synchronou
On 2015-09-03 12:10:08 -0400, Robert Haas wrote:
> On Thu, Sep 3, 2015 at 6:57 AM, Bruce Momjian wrote:
> > Yes, I assumed that. Logical replication uses WAL, so if you are
> > synchronous with WAL, logical replication is synchronous too. However,
> > of course, it is synchronous in being durabl
On Thu, Sep 3, 2015 at 6:57 AM, Bruce Momjian wrote:
> Yes, I assumed that. Logical replication uses WAL, so if you are
> synchronous with WAL, logical replication is synchronous too. However,
> of course, it is synchronous in being durable, not synchronous in terms
> of applying the WAL. This
On Thu, Sep 3, 2015 at 03:40:40PM +0200, Tomas Vondra wrote:
> Not really, the idea is that you don't need to create the replica
> immediately. The system recognizes that primary shard location is
> unavailable and redirects the tasks to the "replicas." So the time
> to recreate the failed node is
Josh Berkus wrote:
>>> You'd also need a way to let the connection nodes know when a replica
>>> has fallen behind so that they can be taken out of
>>> load-balancing/sharding for read queries. For the synchronous model,
>>> that would be "fallen behind at all"; for asynchronous it would be
>>>
Hi,
On 09/03/2015 05:02 AM, Amit Kapila wrote:
On Thu, Sep 3, 2015 at 8:28 AM, Bruce Momjian mailto:br...@momjian.us>> wrote:
>
> On Wed, Sep 2, 2015 at 07:50:25PM -0700, Joshua Drake wrote:
> > >Can you explain why logical replication is better than binary
> > >replication for this use-cas
On Thu, Sep 3, 2015 at 10:33:12AM +0200, Petr Jelinek wrote:
> On 2015-09-02 19:57, Josh Berkus wrote:
> > On 09/01/2015 04:14 PM, Petr Jelinek wrote:
> >> On 2015-09-02 00:09, Josh Berkus wrote:
> >>> Not really, the mechanism is different and the behavior is different.
> >>> One critical deficie
On 2015-09-02 19:57, Josh Berkus wrote:
> On 09/01/2015 04:14 PM, Petr Jelinek wrote:
>> On 2015-09-02 00:09, Josh Berkus wrote:
>>> Not really, the mechanism is different and the behavior is different.
>>> One critical deficiency in using binary syncrep is that you can't do
>>> round-robin redunda
>> One lesson from XL we got is that we need testing framework for cluster,
>> so any cluster project should at least pass functional and performance
>> testing.
>>
>
> +1. In early XC days, we focused a lot on adding newer features and
> supporting as many PG features as possible. That took its t
On Wed, Sep 2, 2015 at 9:04 PM, Oleg Bartunov wrote:
>
>
>
> One lesson from XL we got is that we need testing framework for cluster,
> so any cluster project should at least pass functional and performance
> testing.
>
+1. In early XC days, we focused a lot on adding newer features and
supporti
On Thu, Sep 3, 2015 at 8:28 AM, Bruce Momjian wrote:
>
> On Wed, Sep 2, 2015 at 07:50:25PM -0700, Joshua Drake wrote:
> > >Can you explain why logical replication is better than binary
> > >replication for this use-case?
> > >
> >
> > Selectivity?
>
> I was assuming you would just create identica
On Wed, Sep 2, 2015 at 09:03:25PM -0400, Robert Haas wrote:
> > Can you explain why logical replication is better than binary
> > replication for this use-case?
>
> Uh, well, for the same reasons it is better in many other cases.
> Particularly, you probably don't want to replicate all the data o
On Wed, Sep 2, 2015 at 07:50:25PM -0700, Joshua Drake wrote:
> >Can you explain why logical replication is better than binary
> >replication for this use-case?
> >
>
> Selectivity?
I was assuming you would just create identical slaves to handle failure,
rather than moving selected data around.
On 09/02/2015 03:56 PM, Bruce Momjian wrote:
On Wed, Sep 2, 2015 at 02:41:46PM -0400, Robert Haas wrote:
4. Therefore, I think that we should instead use logical replication,
which might be either synchronous or asynchronous. When you modify
one copy of the data, that change will then be repli
On Thu, Sep 3, 2015 at 3:41 AM, Robert Haas wrote:
> 3. IIUC, Postgres-XC handles this problem by reducing at least
> volatile functions, maybe all functions, to constants. Then it
> generates an SQL statement to be sent to the data node to make the
> appropriate change. If there's more than one
On Wed, Sep 2, 2015 at 6:56 PM, Bruce Momjian wrote:
> On Wed, Sep 2, 2015 at 02:41:46PM -0400, Robert Haas wrote:
>> 4. Therefore, I think that we should instead use logical replication,
>> which might be either synchronous or asynchronous. When you modify
>> one copy of the data, that change w
On Wed, Sep 2, 2015 at 12:03:36PM -0700, Josh Berkus wrote:
> Well, there is a WIP patch for that, which IMHO would be much improved
> by having a concrete use-case like this one. What nobody is working on
> -- and we've vetoed in the past -- is a way of automatically failing and
> removing from
On Wed, Sep 2, 2015 at 02:41:46PM -0400, Robert Haas wrote:
> 4. Therefore, I think that we should instead use logical replication,
> which might be either synchronous or asynchronous. When you modify
> one copy of the data, that change will then be replicated to all other
> nodes. If you are OK
On 09/02/2015 12:30 PM, Robert Haas wrote:
> On Wed, Sep 2, 2015 at 3:03 PM, Josh Berkus wrote:
>>> 4. Therefore, I think that we should instead use logical replication,
>>> which might be either synchronous or asynchronous. When you modify
>>> one copy of the data, that change will then be repli
On 09/02/2015 08:27 PM, Robert Haas wrote:
On Wed, Sep 2, 2015 at 1:59 PM, Merlin Moncure
wrote:
This strikes me as a bit of a conflict of interest with FDW which
seems to want to hide the fact that it's foreign; the FDW
implementation makes it's own optimization decisions which might
make s
On Wed, Sep 2, 2015 at 3:03 PM, Josh Berkus wrote:
>> 4. Therefore, I think that we should instead use logical replication,
>> which might be either synchronous or asynchronous. When you modify
>> one copy of the data, that change will then be replicated to all other
>> nodes. If you are OK with
On 09/02/2015 11:41 AM, Robert Haas wrote:
> On Wed, Sep 2, 2015 at 1:57 PM, Josh Berkus wrote:
>> Even if it's only on paper, any new sharding design needs to address
>> these questions:
>>
>> 1. How do we ensure no/minimal data is lost if we lose a node?
>> 2. How do we replace a lost node (with
On Wed, Sep 2, 2015 at 1:57 PM, Josh Berkus wrote:
> Even if it's only on paper, any new sharding design needs to address
> these questions:
>
> 1. How do we ensure no/minimal data is lost if we lose a node?
> 2. How do we replace a lost node (without taking the cluster down)?
>2. a. how do we
On Wed, Sep 2, 2015 at 1:59 PM, Merlin Moncure wrote:
> On Tue, Sep 1, 2015 at 11:18 AM, Robert Haas wrote:
>> It would be a bad idea to cling blindly to the FDW infrastructure if
>> it's fundamentally inadequate to do what we want. On the other hand,
>> it would also be a bad idea to set about
On Tue, Sep 1, 2015 at 11:18 AM, Robert Haas wrote:
> It would be a bad idea to cling blindly to the FDW infrastructure if
> it's fundamentally inadequate to do what we want. On the other hand,
> it would also be a bad idea to set about recreating it without a
> really good reason, and - just to
On 09/01/2015 04:14 PM, Petr Jelinek wrote:
> On 2015-09-02 00:09, Josh Berkus wrote:
>> On 09/01/2015 02:29 PM, Tomas Vondra wrote:
>>> So while you may be right in single-DC deployments, with multi-DC
>>> deployments the situation is quite different - not only that the network
>>> bandwidth is no
On Tue, Sep 1, 2015 at 7:08 PM, Robert Haas wrote:
> On Tue, Sep 1, 2015 at 12:00 AM, Pavan Deolasee
> wrote:
> > My worry is that if we start implementing them again from scratch, it
> will
> > take a few years before we get them in a usable state. What XC/XL lacked
> is
> > probably a Robert H
On Tue, Sep 1, 2015 at 06:11:45PM -0400, Bruce Momjian wrote:
> Let me clearer about what the Citus Data paper shows. I said originally
> that the data was sent to the coordinator, sorted, then resent to the
> shards, but the document:
>
> https://goo.gl/vJWF85
>
> https://www.citus
On 2015/09/02 20:42, Amit Kapila wrote:
On Wed, Sep 2, 2015 at 4:19 PM, Pavan Deolasee mailto:pavan.deola...@gmail.com>> wrote:
> On Wed, Sep 2, 2015 at 3:55 PM, Amit Langote
mailto:langote_amit...@lab.ntt.co.jp>>
wrote:
>> On 2015-09-02 PM 06:41, Amit Langote wrote:
>> Which, I now realize, i
On Wed, Sep 2, 2015 at 4:19 PM, Pavan Deolasee
wrote:
> On Wed, Sep 2, 2015 at 3:55 PM, Amit Langote <
langote_amit...@lab.ntt.co.jp> wrote:
>>
>> On 2015-09-02 PM 06:41, Amit Langote wrote:
>>
>> Which, I now realize, is not the worry Amit Kapila's expresses.
>>
>> The deadlock was *indeed detect
On Wed, Sep 2, 2015 at 3:55 PM, Amit Langote
wrote:
> On 2015-09-02 PM 06:41, Amit Langote wrote:
> >
> > I think Albe may have a point here...
> >
> > Even inherited updates case appears to cause a deadlock if they are in
> > different queries. Demonstrated below:
> >
> > -- setup
> > CREATE TAB
On 2015-09-02 PM 06:41, Amit Langote wrote:
>
> I think Albe may have a point here...
>
> Even inherited updates case appears to cause a deadlock if they are in
> different queries. Demonstrated below:
>
> -- setup
> CREATE TABLE t(a int);
> CREATE TABLE t1() INHERITS(t);
> CREATE TABLE t2() INH
On Wed, Sep 2, 2015 at 12:49 AM, Josh Berkus wrote:
> On 09/01/2015 11:36 AM, Tomas Vondra wrote:
> >> We want multiple copies of shards created by the sharding system itself.
> >> Having a separate, and completely orthagonal, redundancy system to the
> >> sharding system is overly burdensome o
On 2015-09-02 PM 05:07, Etsuro Fujita wrote:
> On 2015/09/02 16:40, Amit Langote wrote:
>> On 2015-09-02 PM 04:07, Albe Laurenz wrote:
>>>
>>> That would only hold for a single query, right?
>>>
>>> If 1. and 2. in the above example come from different queries within one
>>> transaction, you cannot
Etsuro Fujita wrote:
> On 2015/09/02 16:40, Amit Langote wrote:
>> On 2015-09-02 PM 04:07, Albe Laurenz wrote:
>>> Amit Langote wrote:
On 2015-09-02 PM 03:25, Amit Kapila wrote:
> Will it handle deadlocks across different table partitions. Consider
> a case as below:
>
> T1
>>>
On 2015/09/02 16:40, Amit Langote wrote:
On 2015-09-02 PM 04:07, Albe Laurenz wrote:
Amit Langote wrote:
On 2015-09-02 PM 03:25, Amit Kapila wrote:
Will it handle deadlocks across different table partitions. Consider
a case as below:
T1
1. Updates row R1 of T1 on shard S1
2. Updates row R2 of
On 2015-09-02 PM 04:07, Albe Laurenz wrote:
> Amit Langote wrote:
>> On 2015-09-02 PM 03:25, Amit Kapila wrote:
>>> Will it handle deadlocks across different table partitions. Consider
>>> a case as below:
>>>
>>> T1
>>> 1. Updates row R1 of T1 on shard S1
>>> 2. Updates row R2 of T2 on shard S2
>>
Amit Langote wrote:
> On 2015-09-02 PM 03:25, Amit Kapila wrote:
>> Will it handle deadlocks across different table partitions. Consider
>> a case as below:
>>
>> T1
>> 1. Updates row R1 of T1 on shard S1
>> 2. Updates row R2 of T2 on shard S2
>>
>> T2
>> 1. Updates row R2 of T2 on shard S2
>> 2. U
On 2015/09/02 15:40, Amit Langote wrote:
On 2015-09-02 PM 03:25, Amit Kapila wrote:
On Wed, Sep 2, 2015 at 11:35 AM, Etsuro Fujita
The UPDATE/DELETE pushdown, which I've proposed, would ensure the sane
behaviour for inherited UPDATEs/DELETEs, as existing non-pushed-down
UPDATE/DELETE does, bec
On 2015-09-02 PM 03:25, Amit Kapila wrote:
> On Wed, Sep 2, 2015 at 11:35 AM, Etsuro Fujita
>>
>> The UPDATE/DELETE pushdown, which I've proposed, would ensure the sane
>> behaviour for inherited UPDATEs/DELETEs, as existing non-pushed-down
>> UPDATE/DELETE does, because inheritance_planner guaran
On Wed, Sep 2, 2015 at 11:35 AM, Etsuro Fujita
wrote:
> On 2015/09/02 14:28, Amit Langote wrote:
>
>> On 2015-09-02 PM 01:28, Amit Kapila wrote:
>>
>>> On Tue, Sep 1, 2015 at 9:48 PM, Robert Haas
wrote:
> I'm not averse to making the "connect to the remote nodes" part of
> this
On 2015/09/02 14:28, Amit Langote wrote:
On 2015-09-02 PM 01:28, Amit Kapila wrote:
On Tue, Sep 1, 2015 at 9:48 PM, Robert Haas wrote:
I'm not averse to making the "connect to the remote nodes" part of
this solution use something other than the FDW infrastructure at some
point in time if someb
On 2015-09-02 PM 01:28, Amit Kapila wrote:
>> On Tue, Sep 1, 2015 at 9:48 PM, Robert Haas wrote:
>>>
>>> I'm not averse to making the "connect to the remote nodes" part of
>>> this solution use something other than the FDW infrastructure at some
>>> point in time if somebody's prepared to build so
On Tue, Sep 1, 2015 at 4:25 PM, Bruce Momjian wrote:
>
>
> The document opens a big question --- when queries can't be processed in
> a traditional top/down fashion, Citus has the goal of sending groups of
> results up the the coordinator, reordering them, then sending them back
> to the shards fo
> On Tue, Sep 1, 2015 at 9:48 PM, Robert Haas wrote:
> >
> > On Tue, Sep 1, 2015 at 4:15 AM, Andres Freund
wrote:
> > > On 2015-08-31 20:54:51 -0400, Bruce Momjian wrote:
> > >> Uh, we already have a list of things we need to add to FDWs to make
them
> > >> work, and Citus Data has provided a doc
On 2015-09-02 00:09, Josh Berkus wrote:
> On 09/01/2015 02:29 PM, Tomas Vondra wrote:
>> For example assume that there are just two shards in two separate data
>> centers, connected by a link with limited bandwidth. Now, let's assume
>> you always keep a local replica for failover. So you have A1+A
On Tue, Sep 1, 2015 at 10:17 AM, Robert Haas wrote:
> On Tue, Sep 1, 2015 at 1:06 PM, Josh Berkus wrote:
>> You're assuming that our primary bottleneck for writes is IO. It's not
>> at present for most users, and it certainly won't be in the future. You
>> need to move your thinking on systems
On Tue, Sep 1, 2015 at 12:40:40PM -0400, Robert Haas wrote:
> On Tue, Sep 1, 2015 at 6:55 AM, Bruce Momjian wrote:
> > I assumed these queries were going to be solved by sending as digested
> > data as possible to the coordinator, and having the coordinator complete
> > any remaining processing.
On 09/01/2015 02:29 PM, Tomas Vondra wrote:
> Hi,
>
> On 09/01/2015 09:19 PM, Josh Berkus wrote:
>> Other way around, that is, having replication standbys as the only
>> method of redundancy requires either high data loss or high latency
>> for all writes.
>
> I haven't said that. I said that we
On Sep 1, 2015, at 1:47 PM, Robert Haas wrote:
> Admittedly, there are some problems with snapshots here: if you don't
> do anything special about snapshots, then what you have here will be
> "eventually consistent" behavior. But that might be suitable for some
> environments, such as very loose
On Tue, Sep 1, 2015 at 08:18:38AM -0700, Joshua Drake wrote:
> On 09/01/2015 02:48 AM, Bruce Momjian wrote:
> >On Tue, Sep 1, 2015 at 09:30:41AM +0530, Pavan Deolasee wrote:
>
> >There is no question that using XC/XL will get us to a usable solution
> >faster, but see my recent post to Josh Berk
Hi,
On 09/01/2015 09:19 PM, Josh Berkus wrote:
On 09/01/2015 11:36 AM, Tomas Vondra wrote:
We want multiple copies of shards created by the sharding system
itself. Having a separate, and completely orthagonal, redundancy
system to the sharding system is overly burdensome on the DBA and
makes lo
On Tue, Sep 1, 2015 at 3:19 PM, Josh Berkus wrote:
> Mind you, I see a strong place for binary replication and BDR for
> multi-region redundancy; you really don't want that to be part of the
> sharding system if you're aiming for write scalability.
I mostly agree, but keep in mind that you don't
On 09/01/2015 11:36 AM, Tomas Vondra wrote:
>> We want multiple copies of shards created by the sharding system itself.
>> Having a separate, and completely orthagonal, redundancy system to the
>> sharding system is overly burdensome on the DBA and makes low-data-loss
>> HA impossible.
>
> IMHO
Hi,
On 09/01/2015 08:22 PM, Andres Freund wrote:
On 2015-09-01 14:11:21 -0400, Robert Haas wrote:
On Tue, Sep 1, 2015 at 2:04 PM, Tomas Vondra
wrote:
Memory bandwidth, for example. It's quite difficult to spot, because the
intuition is that memory is fast, but thanks to improvements in storag
Hi,
On 08/31/2015 10:16 PM, Josh Berkus wrote:
It's also important to recognize that there are three major use-cases
for write-scalable clustering:
* OLTP: small-medium cluster, absolute ACID consistency,
bottlnecked on small writes per second
* DW: small-large cluster, ACID optional,
bot
On 2015-09-01 14:11:21 -0400, Robert Haas wrote:
> On Tue, Sep 1, 2015 at 2:04 PM, Tomas Vondra
> wrote:
> > Memory bandwidth, for example. It's quite difficult to spot, because the
> > intuition is that memory is fast, but thanks to improvements in storage (and
> > stagnation in RAM bandwidth), t
On Tue, Sep 1, 2015 at 2:04 PM, Tomas Vondra
wrote:
> Memory bandwidth, for example. It's quite difficult to spot, because the
> intuition is that memory is fast, but thanks to improvements in storage (and
> stagnation in RAM bandwidth), this is becoming a significant issue.
I'd appreciate any ti
Hi,
On 09/01/2015 07:17 PM, Robert Haas wrote:
On Tue, Sep 1, 2015 at 1:06 PM, Josh Berkus wrote:
You're assuming that our primary bottleneck for writes is IO. It's
not at present for most users, and it certainly won't be in the
future. You need to move your thinking on systems resources into
On 09/01/2015 10:17 AM, Robert Haas wrote:
> On Tue, Sep 1, 2015 at 1:06 PM, Josh Berkus wrote:
>> Any sharding solution worth bothering with will solve some or all of the
>> above by extending our ability to process requests across multiple
>> nodes. Any solution which does not is merely an acad
On 09/01/2015 10:06 AM, Josh Berkus wrote:
On 09/01/2015 02:39 AM, Bruce Momjian wrote:
On Mon, Aug 31, 2015 at 01:16:21PM -0700, Josh Berkus wrote:
Our real future bottlenecks are:
* ability to handle more than a few hundred connections
This, 1000 times this. No a connection pooler doesn'
On Tue, Sep 1, 2015 at 1:06 PM, Josh Berkus wrote:
> You're assuming that our primary bottleneck for writes is IO. It's not
> at present for most users, and it certainly won't be in the future. You
> need to move your thinking on systems resources into the 21st century,
> instead of solving the
On 09/01/2015 02:39 AM, Bruce Momjian wrote:
> On Mon, Aug 31, 2015 at 01:16:21PM -0700, Josh Berkus wrote:
>> I'm also going to pontificate that, for a future solution, we should not
>> focus on write *IO*, but rather on CPU and RAM. The reason for this
>> thinking is that, with the latest improve
On Tue, Sep 1, 2015 at 7:01 AM, Mason S wrote:
> For efficient internodes joins with row shipping, FDWs may also not be easy
> to do. Maybe it is possible if we optionally pass in lists of other nodes
> and information about how they are partitioned so data knows where to get
> shipped.
>
> A chal
On Tue, Sep 1, 2015 at 6:55 AM, Bruce Momjian wrote:
> I assumed these queries were going to be solved by sending as digested
> data as possible to the coordinator, and having the coordinator complete
> any remaining processing. I think we are going to need to decide if
> such "sending data back
On 09/01/2015 09:08 AM, Robert Haas wrote:
On Tue, Sep 1, 2015 at 12:00 AM, Pavan Deolasee
From my point of view, and EnterpriseDB's point of view, anything that
doesn't go into the core PostgreSQL distribution isn't really getting
us where we need to be. If there's code in XL that would be
On Tue, Sep 1, 2015 at 4:15 AM, Andres Freund wrote:
> On 2015-08-31 20:54:51 -0400, Bruce Momjian wrote:
>> Uh, we already have a list of things we need to add to FDWs to make them
>> work, and Citus Data has provided a document of more things that are
>> needed, https://goo.gl/vJWF85. I am not
On Tue, Sep 1, 2015 at 12:00 AM, Pavan Deolasee
wrote:
> My worry is that if we start implementing them again from scratch, it will
> take a few years before we get them in a usable state. What XC/XL lacked is
> probably a Robert Haas or a Tom Lane who could look at the work and suggest
> major ed
On Tue, Sep 01, 2015 at 10:15:27AM +0200, Andres Freund wrote:
> On 2015-08-31 20:54:51 -0400, Bruce Momjian wrote:
> > Uh, we already have a list of things we need to add to FDWs to make them
> > work, and Citus Data has provided a document of more things that are
> > needed, https://goo.gl/vJWF85
On 09/01/2015 02:58 AM, Mark Kirkwood wrote:
On 01/09/15 21:41, Bruce Momjian wrote:
I'm thinking that partitioning and sharding are two different things:
Partitioning is about reducing the amount of table data accessed and
also perhaps easing admin activities (archiving/removing old stuff et
On 09/01/2015 02:48 AM, Bruce Momjian wrote:
On Tue, Sep 1, 2015 at 09:30:41AM +0530, Pavan Deolasee wrote:
There is no question that using XC/XL will get us to a usable solution
faster, but see my recent post to Josh Berkus --- the additional code
will be so burdensome that I doubt it would
On Tue, Sep 1, 2015 at 6:55 AM, Bruce Momjian wrote:
> On Mon, Aug 31, 2015 at 11:23:58PM -0300, Alvaro Herrera wrote:
> > Bruce Momjian wrote:
> >
> > > My hope is that many FDW improvements will benefit sharding and
> > > non-sharding workloads, but I bet some improvements are going to be
> > >
On Mon, Aug 31, 2015 at 11:23:58PM -0300, Alvaro Herrera wrote:
> Bruce Momjian wrote:
>
> > My hope is that many FDW improvements will benefit sharding and
> > non-sharding workloads, but I bet some improvements are going to be
> > sharding-specific. I would say we are still in the exploratory s
On Mon, Aug 31, 2015 at 3:08 PM, Alexander Korotkov <
a.korot...@postgrespro.ru> wrote:
> On Mon, Aug 31, 2015 at 9:48 PM, Mason S wrote:
>
>>
>>> We also a bit disappointed by Huawei position about CSN patch, we hoped
>>> to use for our XTM.
>>>
>>
>> Disappointed in what way? Moving to some s
On 01/09/15 21:41, Bruce Momjian wrote:
Well, reworking our partitioning system is one of the things required
for sharding, so at least we will clean up one mess while we create
another. ;-)
Seem my post to Josh Berkus just now --- I think if we don't use FDWs,
that sharding is such a limited
On Tue, Sep 1, 2015 at 09:30:41AM +0530, Pavan Deolasee wrote:
> My worry is that if we start implementing them again from scratch, it will
> take
> a few years before we get them in a usable state. What XC/XL lacked is
> probably
> a Robert Haas or a Tom Lane who could look at the work and sugg
On Tue, Sep 1, 2015 at 10:15:27AM +0200, Andres Freund wrote:
> On 2015-08-31 20:54:51 -0400, Bruce Momjian wrote:
> > Uh, we already have a list of things we need to add to FDWs to make them
> > work, and Citus Data has provided a document of more things that are
> > needed, https://goo.gl/vJWF85
On Mon, Aug 31, 2015 at 01:16:21PM -0700, Josh Berkus wrote:
> I'm also going to pontificate that, for a future solution, we should not
> focus on write *IO*, but rather on CPU and RAM. The reason for this
> thinking is that, with the latest improvements in hardware and 9.5
> improvements, it's inc
On 2015-08-31 20:54:51 -0400, Bruce Momjian wrote:
> Uh, we already have a list of things we need to add to FDWs to make them
> work, and Citus Data has provided a document of more things that are
> needed, https://goo.gl/vJWF85. I am not sure how much bigger a red flag
> you want to confirm that
On Tue, Sep 1, 2015 at 3:17 AM, Robert Haas wrote:
>
>
> It seems to me that sharding consists of (1) breaking your data set up
> into shards, (2) possibly replicating some of those shards onto
> multiple machines, and then (3) being able to access the remote data
> from local queries. As far a
On 2015/09/01 9:54, Bruce Momjian wrote:
On Mon, Aug 31, 2015 at 05:10:11PM -0700, Josh Berkus wrote:
As far as (3) is concerned, why
wouldn't we use the foreign data wrapper interface, and specifically
postgres_fdw? That interface was designed for the explicit purpose of
allowing access to rem
Bruce Momjian wrote:
> My hope is that many FDW improvements will benefit sharding and
> non-sharding workloads, but I bet some improvements are going to be
> sharding-specific. I would say we are still in the exploratory stage,
> but based on the number of people who care about this feature and
On Mon, Aug 31, 2015 at 05:10:11PM -0700, Josh Berkus wrote:
> > As far as (3) is concerned, why
> > wouldn't we use the foreign data wrapper interface, and specifically
> > postgres_fdw? That interface was designed for the explicit purpose of
> > allowing access to remote data sources, and a lot
On Mon, 2015-08-31 at 22:21 +, Robert Haas wrote:
> It seems to me that sharding consists of (1) breaking your data set up
> into shards, (2) possibly replicating some of those shards onto
> multiple machines, and then (3) being able to access the remote data
> from local queries. [...]
I be
On 08/31/2015 02:47 PM, Robert Haas wrote:
> On Mon, Aug 31, 2015 at 4:16 PM, Josh Berkus wrote:
>> First, let me put out there that I think the horizontal scaling project
>> which has buy-in from the community and we're working on is infinitely
>> better than the one we're not working on or is an
Hi Bruce,
Sumedh from Citus Data here.
> August, 2015: While speaking at SFPUG, Citus Data approached me about joining
the FDW sharding team. They have been invited to the September 1 meeting,
as have the XC and XL people.
I'd like to add a clarification. We already tried the FDW APIs for pg_s
On Mon, Aug 31, 2015 at 4:16 PM, Josh Berkus wrote:
> First, let me put out there that I think the horizontal scaling project
> which has buy-in from the community and we're working on is infinitely
> better than the one we're not working on or is an underresourced fork.
> So we're in agreement on
On 08/31/2015 01:16 PM, Josh Berkus wrote:
All, Bruce:
I'm also going to pontificate that, for a future solution, we should not
focus on write *IO*, but rather on CPU and RAM. The reason for this
thinking is that, with the latest improvements in hardware and 9.5
improvements, it's increasingl
All, Bruce:
First, let me put out there that I think the horizontal scaling project
which has buy-in from the community and we're working on is infinitely
better than the one we're not working on or is an underresourced fork.
So we're in agreement on that. However, I think there's a lot of room
f
On Mon, Aug 31, 2015 at 2:12 AM, Oleg Bartunov wrote:
>
> AFAIK, XC/XL has already some customers and that is an additional pressure
> on their development team, which is now called X2. I don't exactly know how
> internal Huawei's MPPDB is connected to XC/XL.
>
Huawei's MPPDB is based on PG-XC an
On Mon, Aug 31, 2015 at 9:48 PM, Mason S wrote:
>
>> We also a bit disappointed by Huawei position about CSN patch, we hoped
>> to use for our XTM.
>>
>
> Disappointed in what way? Moving to some sort of CSN approach seems to
> open things up for different future ideas. In the short term, it wo
On Mon, Aug 31, 2015 at 02:48:31PM -0400, Mason S wrote:
> I assume that future work around PG sharding probably would be more likely to
> be accepted with the FDW approach. One could perhaps work on pushing down
> joins, aggregates and order by, then look at any optimizations gained if code
> is m
>
>
> We also a bit disappointed by Huawei position about CSN patch, we hoped
> to use for our XTM.
>
Disappointed in what way? Moving to some sort of CSN approach seems to open
things up for different future ideas. In the short term, it would mean
replacing potentially large snapshots and longe
1 - 100 of 113 matches
Mail list logo
