Ok. Seems that an upgrade might fix these problems. Is Cassandra 1.x.x stable enough to upgrade for, or should we wait for a couple of weeks?
2012/1/27 Edward Capriolo <edlinuxg...@gmail.com> > I would not say that issuing restart after x days is a good idea. You are > mostly developing a superstition. You should find the source of the > problem. It could be jmx or thrift clients not closing connections. We > don't restart nodes on a regiment they work fine. > > > On Thursday, January 26, 2012, Mike Panchenko <m...@mihasya.com> wrote: > > There are two relevant bugs (that I know of), both resolved in somewhat > recent versions, which make somewhat regular restarts beneficial > > https://issues.apache.org/jira/browse/CASSANDRA-2868 (memory leak in > GCInspector, fixed in 0.7.9/0.8.5) > > https://issues.apache.org/jira/browse/CASSANDRA-2252 (heap > fragmentation due to the way memtables used to be allocated, refactored in > 1.0.0) > > Restarting daily is probably too frequent for either one of those > problems. We usually notice degraded performance in our ancient cluster > after ~2 weeks w/o a restart. > > As Aaron mentioned, if you have plenty of disk space, there's no reason > to worry about "cruft" sstables. The size of your active set is what > matters, and you can determine if that's getting too big by watching for > iowait (due to reads from the data partition) and/or paging activity of the > java process. When you hit that problem, the solution is to 1. try to tune > your caches and 2. add more nodes to spread the load. I'll reiterate - > looking at raw disk space usage should not be your guide for that. > > "Forcing" a gc generally works, but should not be relied upon (note > "suggest" in > http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#gc()). > It's great news that 1.0 uses a better mechanism for releasing unused > sstables. > > nodetool compact triggers a "major" compaction and is no longer a > recommended by datastax (details here > http://www.datastax.com/docs/1.0/operations/tuning#tuning-compactionbottom of > the page). > > Hope this helps. > > Mike. > > On Wed, Jan 25, 2012 at 5:14 PM, aaron morton <aa...@thelastpickle.com> > wrote: > > > > That disk usage pattern is to be expected in pre 1.0 versions. Disk > usage is far less interesting than disk free space, if it's using 60 GB and > there is 200GB thats ok. If it's using 60Gb and there is 6MB free thats a > problem. > > In pre 1.0 the compacted files are deleted on disk by waiting for the > JVM do decide to GC all remaining references. If there is not enough space > (to store the total size of the files it is about to write or compact) on > disk GC is forced and the files are deleted. Otherwise they will get > deleted at some point in the future. > > In 1.0 files are reference counted and space is freed much sooner. > > With regard to regular maintenance, node tool cleanup remvos data from a > node that it is no longer a replica for. This is only of use when you have > done a token move. > > I would not recommend a daily restart of the cassandra process. You will > lose all the run time optimizations the JVM has made (i think the mapped > files pages will stay resident). As well as adding additional entropy to > the system which must be repaired via HH, RR or nodetool repair. > > If you want to see compacted files purged faster the best approach would > be to upgrade to 1.0. > > Hope that helps. > > ----------------- > > Aaron Morton > > Freelance Developer > > @aaronmorton > > http://www.thelastpickle.com > > On 26/01/2012, at 9:51 AM, R. Verlangen wrote: > > > > In his message he explains that it's for " Forcing a GC ". GC stands for > garbage collection. For some more background see: > http://en.wikipedia.org/wiki/Garbage_collection_(computer_science) > > Cheers! > > > > 2012/1/25 <mike...@thomsonreuters.com> > > > > Karl, > > > > Can you give a little more details on these 2 lines, what do they do? > > > > java -jar cmdline-jmxclient-0.10.3.jar - localhost:8080 > > java.lang:type=Memory gc > > > > Thank you, > > Mike > > > > -----Original Message----- > > From: Karl Hiramoto [mailto:k...@hiramoto.org] > > Sent: Wednesday, January 25, 2012 12:26 PM > > To: user@cassandra.apache.org > > Subject: Re: Restart cassandra every X days? > > > > > > On 01/25/12 19:18, R. Verlangen wrote: > >> Ok thank you for your feedback. I'll add these tasks to our daily > >> cassandra maintenance cronjob. Hopefully this will keep things under > >> controll. > > > > I forgot to mention that we found that Forcing a GC also cleans up some > > space. > > > > > > in a cronjob you can do this with > > http://crawler.archive.org/cmdline-jmxclient/ > > > > > > my cron >
