It's interesting topic for me too. How about to add measurement on static disk utilization (% used) and memory utilization ( rss, JVM heap, JVM GC )?
maki From iPhone On 2011/05/12, at 0:49, Tomer B <tomer...@gmail.com> wrote: > Hi > > I'm trying to predict when my cluster would soon be needing new nodes > added, i want a continuous graph telling my of my cluster health so > that when i see my cluster becomes more and more busy (I want numbers > & measurments) i would be able to know i need to start purchasing more > machines and get them into my cluster, so i want to know of that > beforehand. > I'm writing here what I came with after doing some research over net. > I would highly appreciate any additional gauge measurements and ranges > in order to test my cluster health and to know beforehand when i'm > going to soon need more nodes.Although i'm writing down green > gauge,yellow gauge,red gauge, i'm also trying to find a continuous > graph where i can tell where our cluster stand (as much as > possible...) > > Also my recommendation is always before adding new nodes: > > 1. Make sure all nodes are balanced and if not balance them. > 2. Separate commit log drive from data (SSTables) drive > 3. use mmap index only in memory and not auto > 4. Increase disk IO if possible. > 5. Avoid swapping as much as possible. > > > As for my gauge tests for when to add new nodes: > > test: nodetool tpstats -h <cassandra_host> > green gauge: No pending column with number higher > yellow gauge: pending columns 100-2000 > red gauge:Larger than 3000 > > test: iostat -x -n -p -z 5 10 and iostat -xcn 5 > green gauge: kw/s + kr/s reaches is below 25% capacity of disk io > yellow gauge: 20%-50% > red gauge: 50%+ > > test: ostat -x -n -p -z 5 10 and check %b column > green gauge: less than 10% > yellow gauge: 10%-80% > red gauge: 90%+ > > test: nodetool cfstats --host localhost > green gauge: “SSTable count” item does not continually grow over time > yellow gauge: > red gauge: “SSTable count” item continually grows over time > > test: ./nodetool cfstats --host localhost | grep -i pending > green gauge: 0-2 > yellow gauge: 3-100 > red gauge: 101+ > > I would highly appreciate any additional gauge measurements and ranges > in order to test my cluster health and to know ***beforehand*** when > i'm going to soon need more nodes.
