Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-25 Thread Jonathan Lacefield
Sorry to hear about the frustration. How often are you deleting data/what are you setting for ttl on cols? Jonathan Lacefield Solutions Architect, DataStax (404) 822 3487

Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-25 Thread Oleg Dulin
Sigh, so I am back to where I started from... I did lower gc_grace... jmap -histo:live shows heap is stuffed with DeletedColumn and ExpiringColumn This is extremely frustrating. On 2014-03-11 19:24:50 +, Oleg Dulin said: Good news is that since I lowered gc_grace period it collected

Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-11 Thread Oleg Dulin
Good news is that since I lowered gc_grace period it collected over 100Gigs of tombstones and seems much happier now. Oleg On 2014-03-10 13:33:43 +, Jonathan Lacefield said: Hello,   You have several options:   1) going forward lower gc_grace_seconds http://www.datastax.com/documentati

Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-11 Thread Takenori Sato
In addition to the suggestions by Jonathan, you can run a user defined compaction against a particular set of SSTable files, where you want to remove tombstones. But to do that, you need to find such an optimal set. Here you can find a couple of helpful tools. https://github.com/cloudian/support-

Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-10 Thread Keith Wright
Date: Monday, March 10, 2014 at 8:33 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram Hello, You have several options: 1) going forwa

Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-10 Thread Jonathan Lacefield
Hello, You have several options: 1) going forward lower gc_grace_seconds http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configStorage_r.html?pagename=docs&version=1.2&file=configuration/storage_configuration#gc-grace-seconds - this is very use case specific

Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-10 Thread Oleg Dulin
I get that :) What I'd like to know is how to fix that :) On 2014-03-09 20:24:54 +, Takenori Sato said: You have millions of org.apache.cassandra.db.DeletedColumn instances on the snapshot. This means you have lots of column tombstones, and I guess, which are read into memory by slice q

Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-09 Thread Takenori Sato
You have millions of org.apache.cassandra.db.DeletedColumn instances on the snapshot. This means you have lots of column tombstones, and I guess, which are read into memory by slice query. On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin wrote: > I am trying to understand why one of my nodes keeps

need help with Cassandra 1.2 Full GCing -- output of jmap histogram

2014-03-09 Thread Oleg Dulin
I am trying to understand why one of my nodes keeps full GC. I have Xmx set to 8gigs, memtable total size is 2 gigs. Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ -- Regards, Oleg Dulin http://www.olegdulin.com