Ha! I missed the log line: WARN [ReadStage:70] 2014-01-14 13:03:36,963 SliceQueryFilter.java (line 209) Read 1001 live and 1518 tombstoned cells (see tombstone_warn_threshold)
Seems some application code it trying to read a wide row with lots of tombstones. On Thu, Jan 16, 2014 at 8:28 PM, Aaron Morton <aa...@thelastpickle.com>wrote: > c3.4xlarge > > long par new on a machine like this is not normal. > > Do you have a custom comparator or are you using triggers ? > Do you have a data model that creates a lot of tombstones ? > > Try to return the settings to default and then tune from there, that > includes returning to the default JVM GC settings. If for no other reason > than other people will be able to offer advice. > > Have you changed the compaction_throughput ? Put it back if you have. > If you have enabled multi_threaded compaction disable it. > Consider setting concurrent_compactors to 4 or 8 to reduce compaction > churn. > If you have increased in_memory_compaction_limit put it back. > > Cassandra logs > > Can you provide some of the log messages from GCInspector ? How long are > the pauses ? Is there a lot of CMS or ParNew ? > Do you have monitoring in place ? Is CMS able to return the heap to a low > value e.g. < 3Gb ? > > cpu load > 1000% > > Is this all from cassandra ? > try jvmtop (https://code.google.com/p/jvmtop/) to see what cassandra > threads are doing. > > It’s a lot easier to tune a system with fewer non default settings. > > Cheers > > ----------------- > Aaron Morton > New Zealand > @aaronmorton > > Co-Founder & Principal Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > > On 16/01/2014, at 8:22 am, Arya Goudarzi <gouda...@gmail.com> wrote: > > It is not a good idea to change settings without identifying the root > cause. Chances are what you did masked the problem a bit for you, but the > problem is still there, isn't it? > > > On Wed, Jan 15, 2014 at 1:11 AM, Dimetrio <dimet...@flysoft.ru> wrote: > >> I set G1 because GS started to work wrong(dropped messages) with standard >> GC >> settings. >> In my opinion, Cassandra started to work more stable with G1 (it's getting >> less count of timeouts now) but it's not ideally yet. >> I just want cassandra to works fine. >> >> >> >> -- >> View this message in context: >> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-mad-GC-tp7592248p7592257.html >> Sent from the cassandra-u...@incubator.apache.org mailing list archive >> at Nabble.com. >> > > > > -- > Cheers, > -Arya > > > -- Cheers, -Arya
