I also want to point out an issue I filed that was closed as not an issue: CASSANDRA-6654 Basically if you’re using mixed TTLs on columns in a row, the “oldest” TTL is used to determine if tombstones of other columns can be removed. In other words, if you have a column with a 1 day TTL and a column with a 1 year TTL, the 1 day TTLed data (now tombstoned) will not be removed on compaction.
From: Jonathan Lacefield <jlacefi...@datastax.com<mailto:jlacefi...@datastax.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Monday, March 10, 2014 at 8:33 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram Hello, You have several options: 1) going forward lower gc_grace_seconds http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configStorage_r.html?pagename=docs&version=1.2&file=configuration/storage_configuration#gc-grace-seconds - this is very use case specific. Default is 10 days. Some users will put this at 0 for specific use cases. 2) you could also lower tombstone compaction threshold and interval to get tombstone compaction to fire more often on your tables/cfs: https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436 3) to clean out old tombstones you could always run a manual compaction, those these aren't typically recommended though: http://www.datastax.com/documentation/cassandra/1.2/cassandra/tools/toolsNodetool_r.html For 1 and 2, be sure your disks can keep up with compaction to ensure tombstone, or other, compaction fires regularly enough to clean out old tombstones. Also, you probably want to ensure you are using Level Compaction: http://www.datastax.com/dev/blog/when-to-use-leveled-compaction. Again, this assumes your disk system can handle the increased io from Leveled Compaction. Also, you may be running into this with the older version of Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541 Hope this helps. Jonathan Jonathan Lacefield Solutions Architect, DataStax (404) 822 3487 [http://s.c.lnkd.licdn.com/scds/common/u/img/logos/logo_linkedin_92x22.png]<http://www.linkedin.com/in/jlacefield> [http://www.datastax.com/wp-content/themes/datastax-2013/images/email-sig/email-sig-virtual-training.png]<http://www.datastax.com/what-we-offer/products-services/training/virtual-training> On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <oleg.du...@gmail.com<mailto:oleg.du...@gmail.com>> wrote: I get that :) What I'd like to know is how to fix that :) On 2014-03-09 20:24:54 +0000, Takenori Sato said: You have millions of org.apache.cassandra.db.DeletedColumn instances on the snapshot. This means you have lots of column tombstones, and I guess, which are read into memory by slice query. On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <oleg.du...@gmail.com<mailto:oleg.du...@gmail.com>> wrote: I am trying to understand why one of my nodes keeps full GC. I have Xmx set to 8gigs, memtable total size is 2 gigs. Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ -- Regards, Oleg Dulin http://www.olegdulin.com -- Regards, Oleg Dulin http://www.olegdulin.com
