Ok, seems that it's clear what I should do next ;-) 2012/1/28 aaron morton <aa...@thelastpickle.com>
> There are no blockers to upgrading to 1.0.X. > > A > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 28/01/2012, at 7:48 AM, R. Verlangen wrote: > > Ok. Seems that an upgrade might fix these problems. Is Cassandra 1.x.x > stable enough to upgrade for, or should we wait for a couple of weeks? > > 2012/1/27 Edward Capriolo <edlinuxg...@gmail.com> > >> I would not say that issuing restart after x days is a good idea. You are >> mostly developing a superstition. You should find the source of the >> problem. It could be jmx or thrift clients not closing connections. We >> don't restart nodes on a regiment they work fine. >> >> >> On Thursday, January 26, 2012, Mike Panchenko <m...@mihasya.com> wrote: >> > There are two relevant bugs (that I know of), both resolved in somewhat >> recent versions, which make somewhat regular restarts beneficial >> > https://issues.apache.org/jira/browse/CASSANDRA-2868 (memory leak in >> GCInspector, fixed in 0.7.9/0.8.5) >> > https://issues.apache.org/jira/browse/CASSANDRA-2252 (heap >> fragmentation due to the way memtables used to be allocated, refactored in >> 1.0.0) >> > Restarting daily is probably too frequent for either one of those >> problems. We usually notice degraded performance in our ancient cluster >> after ~2 weeks w/o a restart. >> > As Aaron mentioned, if you have plenty of disk space, there's no reason >> to worry about "cruft" sstables. The size of your active set is what >> matters, and you can determine if that's getting too big by watching for >> iowait (due to reads from the data partition) and/or paging activity of the >> java process. When you hit that problem, the solution is to 1. try to tune >> your caches and 2. add more nodes to spread the load. I'll reiterate - >> looking at raw disk space usage should not be your guide for that. >> > "Forcing" a gc generally works, but should not be relied upon (note >> "suggest" in >> http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#gc()). >> It's great news that 1.0 uses a better mechanism for releasing unused >> sstables. >> > nodetool compact triggers a "major" compaction and is no longer a >> recommended by datastax (details here >> http://www.datastax.com/docs/1.0/operations/tuning#tuning-compactionbottom >> of the page). >> > Hope this helps. >> > Mike. >> > On Wed, Jan 25, 2012 at 5:14 PM, aaron morton <aa...@thelastpickle.com> >> wrote: >> > >> > That disk usage pattern is to be expected in pre 1.0 versions. Disk >> usage is far less interesting than disk free space, if it's using 60 GB and >> there is 200GB thats ok. If it's using 60Gb and there is 6MB free thats a >> problem. >> > In pre 1.0 the compacted files are deleted on disk by waiting for the >> JVM do decide to GC all remaining references. If there is not enough space >> (to store the total size of the files it is about to write or compact) on >> disk GC is forced and the files are deleted. Otherwise they will get >> deleted at some point in the future. >> > In 1.0 files are reference counted and space is freed much sooner. >> > With regard to regular maintenance, node tool cleanup remvos data from >> a node that it is no longer a replica for. This is only of use when you >> have done a token move. >> > I would not recommend a daily restart of the cassandra process. You >> will lose all the run time optimizations the JVM has made (i think the >> mapped files pages will stay resident). As well as adding additional >> entropy to the system which must be repaired via HH, RR or nodetool repair. >> > If you want to see compacted files purged faster the best approach >> would be to upgrade to 1.0. >> > Hope that helps. >> > ----------------- >> > Aaron Morton >> > Freelance Developer >> > @aaronmorton >> > http://www.thelastpickle.com >> > On 26/01/2012, at 9:51 AM, R. Verlangen wrote: >> > >> > In his message he explains that it's for " Forcing a GC ". GC stands >> for garbage collection. For some more background see: >> http://en.wikipedia.org/wiki/Garbage_collection_(computer_science) >> > Cheers! >> > >> > 2012/1/25 <mike...@thomsonreuters.com> >> > >> > Karl, >> > >> > Can you give a little more details on these 2 lines, what do they do? >> > >> > java -jar cmdline-jmxclient-0.10.3.jar - localhost:8080 >> > java.lang:type=Memory gc >> > >> > Thank you, >> > Mike >> > >> > -----Original Message----- >> > From: Karl Hiramoto [mailto:k...@hiramoto.org] >> > Sent: Wednesday, January 25, 2012 12:26 PM >> > To: user@cassandra.apache.org >> > Subject: Re: Restart cassandra every X days? >> > >> > >> > On 01/25/12 19:18, R. Verlangen wrote: >> >> Ok thank you for your feedback. I'll add these tasks to our daily >> >> cassandra maintenance cronjob. Hopefully this will keep things under >> >> controll. >> > >> > I forgot to mention that we found that Forcing a GC also cleans up some >> > space. >> > >> > >> > in a cronjob you can do this with >> > http://crawler.archive.org/cmdline-jmxclient/ >> > >> > >> > my cron >> > > >
