Haha Ok. It is not a total waste, but practically your time is better spent in other places. The problem is just about everything is a moving target, schema, request rate, hardware. Generally tuning nudges a couple variables in one direction or the other and you see some decent returns. But each nudge takes a restart and a warm up period, and with how Cassandra distributes requests you likely have to flip several nodes or all of them before you can see the change! By the time you do that its probably a different day or week. Essentially finding our if one setting is better then the other is like a 3 day test in production.
Before c* I used to deal with this in tomcat. Once in a while we would get a dev that read some article about tuning, something about a new jvm, or collector. With bright eyed enthusiasm they would want to try tuning our current cluster. They spend a couple days and measure something and say it was good "lower memory usage". Meanwhile someone else would come to me and say "higher 95th response time". More short pauses, fewer long pauses, great taste, less filing. Most people just want to roflscale their huroku cloud. Tuning stuff is sysadmin work and the cloud has taught us that the cost of sysadmins are needless waste of money. Just kidding ! But I do believe the default cassandra settings are reasonable and typically I find that most who look at tuning GC usually need more hardware and actually need to be tuning something somewhere else. G1 is the perfect example of a time suck. Claims low pause latency for big heaps, and delivers something regarded by the Cassandra community (and hbase as well) that works worse then CMS. If you spent 3 hours switching tuning knobs and analysing, that is 3 hours of your life you will never get back. Better to let SUN and other people worry about tuning (at least from where I sit) On Saturday, September 15, 2012, Peter Schuller <peter.schul...@infidyne.com> wrote: >> Generally tuning the garbage collector is a waste of time. > > Sorry, that's BS. It can be absolutely critical, when done right, and > only "useless" when done wrong. There's a spectrum in between. > >> Just follow >> someone else's recommendation and use that. > > No, don't. > > Most recommendations out there are completely useless in the general > case because someone did some very specific benchmark under very > specific circumstances and then recommends some particular combination > of options. In order to understand whether a particular recommendation > applies to you, you need to know enough about your use-case that I > suspect you're better of just reading up on the available options and > figuring things out. Of course, randomly trying various different > settings to see which seems to work well may be realistic - but you > loose predictability (in the face of changing patterns of traffic for > example) if you don't know why it's behaving like it is. > > If you care about GC related behavior you want to understand how the > application behaves, how the garbage collector behaves, what your > requirements are, and select settings based on those requirements and > how the application and GC behavior combine to produce emergent > behavior. The "best" GC options may vary *wildly* depending on the > nature of your cluster and your goals. There are also non-GC settings > (in the specific case of Cassandra) that affect the interaction with > the garbage collector, like whether you're using row/key caching, or > things like phi conviction threshold and/or timeouts. It's very hard > for anyone to give generalized recommendations. If it weren't, > Cassandra would ship with The One True set of settings that are always > the best and there would be no discussion. > > It's very unfortunate that the state of GC in the freely available > JVM:s is at this point given that there exists known and working > algorithms (and at least one practical implementation) that avoids it, > mostly. But, it's the situation we're in. The only way around it that > I know of if you're on Hotspot, is to have the application behave in > such a way that it avoids the causes of un-predictable behavior w.r.t. > GC by being careful about it's memory allocation and *retention* > profile. For the specific case of avoiding *ever* seeing a full gc, it > gets even more complex. > > -- > / Peter Schuller (@scode, http://worldmodscode.wordpress.com) >
