Re: Upgrade Limitations Question

2015-09-23 Thread Robert Coli
On Wed, Sep 16, 2015 at 7:02 AM, Vasileios Vlachos < vasileiosvlac...@gmail.com> wrote: > > At the end we had to wait for the upgradesstables ti finish on every node. > Just to eliminate the possibility of this being the reason of any weird > behaviour after the upgrade. However, this process migh

Re: Do vnodes need more memory?

2015-09-23 Thread Robert Coli
On Wed, Sep 23, 2015 at 7:09 AM, Tom van den Berge < tom.vandenbe...@gmail.com> wrote: > So it seems that Cassandra simply doesn't have enough memory. I'm trying > to understand if this can be caused by the use of vnodes? Is there an > sensible reason why vnodes would consume more memory than regu

Re: Huge amounts of hinted handoffs for counter table

2015-09-23 Thread Robert Coli
On Wed, Sep 23, 2015 at 7:28 AM, Björn Hachmann wrote: > Today I realized that one of the nodes in our Cassandra cluster (2.1.7) is > storing a lot of hints (>80GB) and I fail to see a convincing way to deal > with them. > ... > We had a look into the table system.hints and from there we learnt t

Re: Huge amounts of hinted handoffs for counter table

2015-09-23 Thread Venkatesh Arivazhagan
What is your replication factor and write consistency? :) On Sep 23, 2015 7:28 AM, "Björn Hachmann" wrote: > Today I realized that one of the nodes in our Cassandra cluster (2.1.7) is > storing a lot of hints (>80GB) and I fail to see a convincing way to deal > with them. > > From the system.log:

Huge amounts of hinted handoffs for counter table

2015-09-23 Thread Björn Hachmann
Today I realized that one of the nodes in our Cassandra cluster (2.1.7) is storing a lot of hints (>80GB) and I fail to see a convincing way to deal with them. >From the system.log: INFO [ScheduledTasks:1] 2015-09-23 14:27:06,692 StatusLogger.java:115 - system.hints 276,10109

Re: Do vnodes need more memory?

2015-09-23 Thread Tom van den Berge
nodetool gcstat tells me this (the Total GC Elapsed is half or more of the Interval). We had to take the production load off the new vnode DC, since it was messing things up badly. It means I'm not able to run any tools against it at the moment. The env.sh is default, and the servers have 8G ram.

Re: Do vnodes need more memory?

2015-09-23 Thread Sebastian Estevez
This is interesting, where are you seeing that you're collecting 50% of the time? Is your env.sh the default? How much ram? Also, can you run this tool and send a minute worth of thread info: wget https://bintray.com/artifact/download/aragozin/generic/sjk-plus-0.3.6.jar java -jar sjk-plus-0.3.6.j

Do vnodes need more memory?

2015-09-23 Thread Tom van den Berge
I have two data centers, each with the same number of nodes, same hardware (CPUs, memory), Cassandra version (2.1.6), replication factory, etc. The only difference it that one data center uses vnodes, and the other doesn't. The non-vnode DC works fine (and has been for a long time) under productio

Re: Cassandra Summit 2015 Roll Call!

2015-09-23 Thread Sebastian Estevez
Hey guys, find me at the Startups booth! Looking forward to meeting some of you in person :) On Sep 22, 2015 8:44 PM, "Steve Robenalt" wrote: > I am here. Wearing my assorted Cassandra shirts from meetups and > conferences. Would be happy to meet anyone from this mailing list because > the conver

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

2015-09-23 Thread Marcelo Valle (BLOOMBERG/ LONDON)
I think there is a very important point in Scylladb - latency. Performance can be an important requirement, but the fact scylladb is written in C and uses lock free algorithms inside means it should have lower latency than Cassandra, which enables it's use for a wider range of applications. It

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

2015-09-23 Thread Peter Lin
Looking at the architecture and what scylladb does, I'm not surprised they got 10x improvement. SeaStar skips a lot of the overhead of copying stuff and it gives them CPU core affinity. Anyone that's listened to Clif Click talk about cache misses, locks and other low level stuff would recognize the

RE: High read latency

2015-09-23 Thread Leleu Eric
For read heavy workload, JVM GC can cause latency issue. (see http://tech.shift.com/post/74311817513/cassandra-tuning-the-jvm-for-read-heavy-workloads) If you have frequent minor GC taking 400ms, it may increase your read latency. Eric De : Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]

JBOD and system keyspaces

2015-09-23 Thread Stuart Bishop
Hi. I'm setting up a new cluster (DSE 4.7), and interested in failure modes of JBOD setups. My main concern is the likelihood of needing to rebuild an entire node whenever a single drive goes wobbly. After I have replaced a failed or failing disk, how likely is it that it contained critical inform