Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-05 Thread Keith Fiske
On Thu, Jun 5, 2014 at 2:09 PM, Sébastien Lorion wrote: > On Thu, Jun 5, 2014 at 12:55 PM, Francisco Olarte > wrote: > >> Hi Sébastien: >> >> On Thu, Jun 5, 2014 at 5:41 PM, Sébastien Lorion >> wrote: >> >> > Correct me if I am wrong, but will it not also suffer the same >> > limitation as

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-05 Thread Sébastien Lorion
On Thu, Jun 5, 2014 at 12:55 PM, Francisco Olarte wrote: > Hi Sébastien: > > On Thu, Jun 5, 2014 at 5:41 PM, Sébastien Lorion > wrote: > > > Correct me if I am wrong, but will it not also suffer the same > > limitation as any statement based replication, namely that the "merged" > > slave w

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-05 Thread Francisco Olarte
Hi Sébastien: On Thu, Jun 5, 2014 at 5:41 PM, Sébastien Lorion wrote: > Correct me if I am wrong, but will it not also suffer the same > limitation as any statement based replication, namely that the "merged" > slave will have to sustain the same write load as all shards combined ? I canno

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-05 Thread Sébastien Lorion
On Wed, Jun 4, 2014 at 1:50 PM, Keith Fiske wrote: > Not sure if this will work for you, but sharing a similar scenario in case > it may work for you. > > An extension I wrote provides similar logical replication as you've > probably seen in other tools. > https://github.com/omniti-labs/mimeo > O

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-04 Thread Koichi Suzuki
Here's PostgreSQL-based sharding solution which provides both read/write horizontal scalability. http://sourceforge.net/apps/mediawiki/postgres-xc/index.php?title=Main_Page http://sourceforge.net/projects/postgres-xc/ Hope this helps. --- Koichi Suzuki 2014-06-03 3:47 GMT+09:00 Sébastien Lorion

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-04 Thread Keith Fiske
On Mon, Jun 2, 2014 at 2:47 PM, Sébastien Lorion wrote: > On Mon, Jun 2, 2014 at 12:52 PM, Kevin Goess wrote: > >> > So my conclusion is that for now, the best way to scale read-only >> queries for a sharded master is to >> > implement map-reduce at the application level. >> >> That's the conclu

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-02 Thread Sébastien Lorion
On Mon, Jun 2, 2014 at 12:52 PM, Kevin Goess wrote: > > So my conclusion is that for now, the best way to scale read-only > queries for a sharded master is to > > implement map-reduce at the application level. > > That's the conclusion I would expect. It's the price you pay for sharding, > it's p

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-02 Thread Kevin Goess
> So my conclusion is that for now, the best way to scale read-only queries for a sharded master is to > implement map-reduce at the application level. That's the conclusion I would expect. It's the price you pay for sharding, it's part of the deal. But it's also the benefit you get from sharding

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-01 Thread John R Pierce
On 6/1/2014 12:59 PM, Sébastien Lorion wrote: it would have been nice to avoid the additional complexity if it had been possible to merge sharded tables on a binary level (which should be much faster than statement level), given that their records will never overlap (i.e. the same record is nev

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-01 Thread Sébastien Lorion
On Thu, May 29, 2014 at 12:58 PM, Sébastien Lorion wrote: > I have a master database sharded by user_id, with globally unique IDs for > everything, except shared configuration data stored in global tables > (resources strings, system parameters, etc). > > What would be the best (ie both fast and