did you check jmx to see if a compaction is going on?
On Mon, Mar 22, 2010 at 5:14 PM, Todd Burruss <bburr...@real.com> wrote: > after running my cluster for a while performance has become unacceptable, > 200+ ms for reads. if running well, i see reads <10ms. when i run iostat > the disk is being hammered by reads. seems like i/o caching isn't even > being used > > avg-cpu: %user %nice %system %iowait %steal %idle > 2.81 0.00 1.41 13.62 0.00 82.16 > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz > avgqu-sz await svctm %util > cciss/c0d0p1 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 > cciss/c0d1p1 > 0.00 0.00 848.50 0.00 13.66 0.00 32.98 > 21.50 25.23 1.18 100.05 > > i run top and i see cassandra's memory usage as follows: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 31510 bburruss 19 0 359g 37g 27g S 48.8 80.1 2137:30 java > > > i set -Xmx10g so it isn't java using the memory. is it mmap i/o? what > would be causing the huge memory usage? > it seems reasonable that the performance is bad because the i/o cache can't > be used properly. >
