.... any ideas or advises? On Mon, Jun 22, 2015 at 10:55 AM, Arun Chaitanya <chaitan64a...@gmail.com> wrote:
> Hello All, > > Now we settled on the following approach. I want to know if there are any > problems that you foresee in the production environment. > > Our Approach: Use Off Heap Memory > > Modifications to default cassandra.yaml and cassandra-env.sh > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > * memory_allocator: JEMallocAllocator > (https://issues.apache.org/jira/browse/CASSANDRA-7883) > * memtable_allocation_type: offheap_objects > > By above two, the slab allocation > (https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires > 1MB heap memory per table, is disabled. The memory for table metadata, > caches and memtable are thus > allocated natively and does not affect GC performance. > > * tombstone_failure_threshold: 100000000 > > Without this, C* throws TombstoneOverwhelmingException while in startup. > This setting looks problematic so I want to know why just creating tables > makes so many tombstones ... > > * -XX:+UseG1GC > > It is good for reducing GC time. > Without this, full GCs > 1s are observed. > > We created 5000 column families with about 1000 entries per column family. > The read/write performance seems to stable. > The problem we saw is with startup time. > > Cassandra Start Time (s) 20 > > > > 349 Average CPU Usage (%) 40 > > > > 49.65 GC Actitivy (%) 2.6 > > > > 0.6 > Thanks a lot in advance. > > On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson <gra...@vast.com> wrote: > >> > I strongly advise against this approach. >>> Jon, I think so too. But so you actually foresee any problems with this >>> approach? >>> I can think of a few. [I want to evaluate if we can live with this >>> problem] >>> >>> >>> Just to be clear, I’m not saying this is a great approach, I AM saying >> that it may be better than having 10000+ CFs, which was the original >> question (it really depends on the use case which wasn’t well defined)… map >> size limit may be a problem, and then there is the CQL vs thrift question >> which could start a flame war; ideally CQL maps should give you the same >> flexibility as arbitrary thrift columns >> >> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad <j...@jonhaddad.com> wrote: >> >> > Sorry for this naive question but how important is this tuning? Can >> this have a huge impact in production? >> >> Massive. Here's a graph of when we did some JVM tuning at my previous >> company: >> >> >> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png >> >> About an order of magnitude difference in performance. >> >> Jon >> >> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <chaitan64a...@gmail.com> >> wrote: >> >>> Thanks Jon and Jack, >>> >>> > I strongly advise against this approach. >>> Jon, I think so too. But so you actually foresee any problems with this >>> approach? >>> I can think of a few. [I want to evaluate if we can live with this >>> problem] >>> >>> - No more CQL. >>> - No data types, everything needs to be a blob. >>> - Limited clustering Keys and default clustering order. >>> >>> > First off, different workloads need different tuning. >>> Sorry for this naive question but how important is this tuning? Can >>> this have a huge impact in production? >>> >>> > You might want to consider a model where you have an application layer >>> that maps logical tenant tables into partition keys within a single large >>> Casandra table, or at least a relatively small number of Cassandra tables. >>> It will depend on the typical size of your tenant tables - very small ones >>> would make sense within a single partition, while larger ones should have >>> separate partitions for a tenant's data. The key here is that tables are >>> expensive, but partitions are cheap and scale very well with Cassandra. >>> We are actually trying similar approach. But we don't want to expose >>> this to application layer. We are attempting to hide this and provide an >>> API. >>> >>> > Finally, you said "10 clusters", but did you mean 10 nodes? You might >>> want to consider a model where you do indeed have multiple clusters, where >>> each handles a fraction of the tenants, since there is no need for separate >>> tenants to be on the same cluster. >>> I meant 10 clusters. We want to split our tables across multiple >>> clusters if above approach is not possible. [But it seems to be very costly] >>> >>> Thanks, >>> >>> >>> >>> >>> >>> >>> >>> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky < >>> jack.krupan...@gmail.com> wrote: >>> >>>> How big is each of the tables - are they all fairly small or fairly >>>> large? Small as in no more than thousands of rows or large as in tens of >>>> millions or hundreds of millions of rows? >>>> >>>> Small tables are are not ideal for a Cassandra cluster since the rows >>>> would be spread out across the nodes, even though it might make more sense >>>> for each small table to be on a single node. >>>> >>>> You might want to consider a model where you have an application layer >>>> that maps logical tenant tables into partition keys within a single large >>>> Casandra table, or at least a relatively small number of Cassandra tables. >>>> It will depend on the typical size of your tenant tables - very small ones >>>> would make sense within a single partition, while larger ones should have >>>> separate partitions for a tenant's data. The key here is that tables are >>>> expensive, but partitions are cheap and scale very well with Cassandra. >>>> >>>> Finally, you said "10 clusters", but did you mean 10 nodes? You might >>>> want to consider a model where you do indeed have multiple clusters, where >>>> each handles a fraction of the tenants, since there is no need for separate >>>> tenants to be on the same cluster. >>>> >>>> >>>> -- Jack Krupansky >>>> >>>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya < >>>> chaitan64a...@gmail.com> wrote: >>>> >>>>> Good Day Everyone, >>>>> >>>>> I am very happy with the (almost) linear scalability offered by C*. We >>>>> had a lot of problems with RDBMS. >>>>> >>>>> But, I heard that C* has a limit on number of column families that can >>>>> be created in a single cluster. >>>>> The reason being each CF stores 1-2 MB on the JVM heap. >>>>> >>>>> In our use case, we have about 10000+ CF and we want to support >>>>> multi-tenancy. >>>>> (i.e 10000 * no of tenants) >>>>> >>>>> We are new to C* and being from RDBMS background, I would like to >>>>> understand how to tackle this scenario from your advice. >>>>> >>>>> Our plan is to use Off-Heap memtable approach. >>>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1 >>>>> >>>>> Each node in the cluster has following configuration >>>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap) >>>>> IMO, this should be able to support 1000 CF with no(very less) impact >>>>> on performance and startup time. >>>>> >>>>> We tackle multi-tenancy using different keyspaces.(Solution I found on >>>>> the web) >>>>> >>>>> Using this approach we can have 10 clusters doing the job. (We >>>>> actually are worried about the cost) >>>>> >>>>> Can you please help us evaluate this strategy? I want to hear >>>>> communities opinion on this. >>>>> >>>>> My major concerns being, >>>>> >>>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting >>>>> 1000 CF right? >>>>> >>>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the >>>>> number of column families increase even when we use multiple keyspace. >>>>> >>>>> 3. I understand the complexity using multi-cluster for single >>>>> application. The code base will get tightly coupled with infrastructure. >>>>> Is >>>>> this the right approach? >>>>> >>>>> Any suggestion is appreciated. >>>>> >>>>> Thanks, >>>>> Arun >>>>> >>>> >>>> >>> >> >
