Re: Cassandra mad GC

2014-01-20 Thread Dimetrio
btw, cassandra cluster is more stable with turned off multithread compaction. One node have more keys than other nodes normal node Keyspace: Social Read Count: 65530294 Read Latency: 2.010432367020969 ms. Write Count: 183948607 Write Latency: 0.04994240148825917 ms.

Re: Cassandra mad GC

2014-01-20 Thread Dimetrio
no triggers, no custom comparators i have a data model that creates a lot of tombstones (users home timeline with many inserts and deletes). how can i reduce tombstones count it this case? >Is this all from cassandra ? yes, with multithread compaction (for c3.4 - 6 threads), compaction_through

Re: Cassandra mad GC

2014-01-20 Thread Dimetrio
I think "Read 1001 live and 1518" is not too many tombstones and its normal -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-mad-GC-tp7592248p7592297.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabb

Re: Cassandra mad GC

2014-01-16 Thread Arya Goudarzi
Ha! I missed the log line: WARN [ReadStage:70] 2014-01-14 13:03:36,963 SliceQueryFilter.java (line 209) Read 1001 live and 1518 tombstoned cells (see tombstone_warn_threshold) Seems some application code it trying to read a wide row with lots of tombstones. On Thu, Jan 16, 2014 at 8:28 PM, Aar

Re: Cassandra mad GC

2014-01-16 Thread Aaron Morton
> c3.4xlarge long par new on a machine like this is not normal. Do you have a custom comparator or are you using triggers ? Do you have a data model that creates a lot of tombstones ? Try to return the settings to default and then tune from there, that includes returning to the default JVM GC

Re: Cassandra mad GC

2014-01-15 Thread Arya Goudarzi
It is not a good idea to change settings without identifying the root cause. Chances are what you did masked the problem a bit for you, but the problem is still there, isn't it? On Wed, Jan 15, 2014 at 1:11 AM, Dimetrio wrote: > I set G1 because GS started to work wrong(dropped messages) with s

RE: Cassandra mad GC

2014-01-15 Thread Dimetrio
I set G1 because GS started to work wrong(dropped messages) with standard GC settings. In my opinion, Cassandra started to work more stable with G1 (it's getting less count of timeouts now) but it's not ideally yet. I just want cassandra to works fine. -- View this message in context: http://c

RE: Cassandra mad GC

2014-01-15 Thread Viktor Jevdokimov
Forgot to ask, what do you want to achieve by changing default GC settings? -Original Message- From: Viktor Jevdokimov [mailto:viktor.jevdoki...@adform.com] Sent: Wednesday, January 15, 2014 10:18 AM To: user@cassandra.apache.org Subject: RE: Cassandra mad GC Simply don't use G1 G

RE: Cassandra mad GC

2014-01-15 Thread Viktor Jevdokimov
Simply don't use G1 GC, it will not be better on Cassandra than CMS, it could be worse. Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063 Fax: +370 5 261 0453 J. Jasinskio 16C, LT-03163 Vilnius, Lithuania Disclaimer: The

Re: Cassandra mad GC

2014-01-14 Thread Arya Goudarzi
Hi, I sympathize with your issue. I recommend adding the following to your JVM flags: JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails" JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps" JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC" JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution" JVM_OPTS="$JVM_OPTS -XX:+PrintGCAp

Re: Cassandra mad GC

2014-01-14 Thread Dimetrio
iostat is clean vm.max_map_count = 131072 -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-mad-GC-tp7592248p7592251.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: Cassandra mad GC

2014-01-14 Thread Dimetrio
JNA is installed java -version java version "1.7.0_45" Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) cassandra 2.0.4, vnodes LCS, LZ4 Compression some cassandra config params: -- View this message in context: http://cas

Re: Cassandra mad GC

2014-01-14 Thread Johnny Miller
Have you followed the recommended prod settings and installed JNA, Oracle JDK etc..? http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/install/installRecommendSettings.html On 14 Jan 2014, at 13:16, Dimetrio wrote: > Hi all. > I have many GC freezes on my cassan