On Tue, Dec 30, 2014 at 3:12 PM, Mikhail Strebkov <streb...@gmail.com> wrote:
> We have a table in our production Cassandra that is stored on 11369 > SSTables. The average SSTable count for the other tables is around 15, and > the read latency for them is much smaller. > Unthrottle compaction, that's an insane number of SSTables. > I tried to run manual compaction (nodetool compact my_keyspace my_table) > but then the node starts spending ~90% of the time in GC and compaction > advances super slowly (it would take a couple of weeks to finish). I > checked IO stats with "iotop" and there is almost no IO going on. > Are you using LCS or STS compaction? > We're running Cassandra on EC2 (m1.xlarge) which has 15G of memory, using > DataStax Community AMI. Our Cassandra version is 2.1.2. We didn't change > Cassandra configuration from the default in the AMI, so Cassandra > calculated 3760M for the heap size. > One solution would be to temporarily increase heap, though going above 8gb or so will increase duration of GCs approaching seconds. Another alternative is to use the JMX call userDefinedCompaction to do a compaction that is less major. Why does Cassandra fall into this "90% CPU time in GC" state and how can I > tune Cassandra so that it can finish the compaction successfully? > Because 12,000 sstables use a lot of heap, and you only have ~4gb of heap. You go into "GC pre-fail" because you can't reclaim enough heap. =Rob http://twitter.com/rcolidba
