Sigh, so I am back to where I started from...
I did lower gc_grace...
jmap -histo:live shows heap is stuffed with DeletedColumn and
ExpiringColumn....
This is extremely frustrating.
On 2014-03-11 19:24:50 +0000, Oleg Dulin said:
Good news is that since I lowered gc_grace period it collected over
100Gigs of tombstones and seems much happier now.
Oleg
On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said:
Hello,
You have several options:
1) going forward lower gc_grace_seconds
http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configStorage_r.html?pagename=docs&version=1.2&file=configuration/storage_configuration#gc-grace-seconds
- this is very use case specific. Default is 10 days. Some
users will put this at 0 for specific use cases.
2) you could also lower tombstone compaction threshold and interval
to get tombstone compaction to fire more often on your tables/cfs:
https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436
3) to clean out old tombstones you could always run a manual
compaction, those these aren't typically recommended though:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/tools/toolsNodetool_r.html
For 1 and 2, be sure your disks can keep up with compaction to ensure
tombstone, or other, compaction fires regularly enough to clean out old
tombstones. Also, you probably want to ensure you are using Level
Compaction:
http://www.datastax.com/dev/blog/when-to-use-leveled-compaction.
Again, this assumes your disk system can handle the increased io from
Leveled Compaction.
Also, you may be running into this with the older version of
Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541
Hope this helps.
Jonathan
Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487
<image>
<image>
On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <oleg.du...@gmail.com> wrote:
I get that :)
What I'd like to know is how to fix that :)
On 2014-03-09 20:24:54 +0000, Takenori Sato said:
You have millions of org.apache.cassandra.db.DeletedColumn instances on
the snapshot.
This means you have lots of column tombstones, and I guess, which are
read into memory by slice query.
On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <oleg.du...@gmail.com> wrote:
I am trying to understand why one of my nodes keeps full GC.
I have Xmx set to 8gigs, memtable total size is 2 gigs.
Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ
--
Regards,
Oleg Dulin
http://www.olegdulin.com
--
Regards,
Oleg Dulin
http://www.olegdulin.com
S
--
Regards,
Oleg Dulin
http://www.olegdulin.com
