Hello.
Were encountering some high read latency issues. But our main Cass expert
is out-of-office so it falls to me.
We're more read than write, though there doesn't seem to be many pending
reads.
I have seen active/pending row-read at three or four, though.
Pool Name Active Pending Completed
FILEUTILS-DELETE-POOL 0 0 46
STREAM-STAGE 0 0 0
RESPONSE-STAGE 0 0 17471880
ROW-READ-STAGE 1 1 37652361
LB-OPERATIONS 0 0 0
MISCELLANEOUS-POOL 0 0 0
GMFD 0 0 154630
LB-TARGET 0 0 0
CONSISTENCY-MANAGER 0 0 2993464
ROW-MUTATION-STAGE 0 0 16383305
MESSAGE-STREAMING-POOL 0 0 0
LOAD-BALANCER-STAGE 0 0 0
FLUSH-SORTER-POOL 0 0 0
MEMTABLE-POST-FLUSHER 0 0 116
FLUSH-WRITER-POOL 0 0 116
AE-SERVICE-STAGE 0 0 0
HINTED-HANDOFF-POOL 0 0 16
Does the high iops on our data mean we need to tune Key or other caches?
$ iostat
Linux 2.6.18-194.11.3.el5 02/03/2011
avg-cpu: %user %nice %system %iowait %steal %idle
1.00 0.00 0.25 1.32 0.00 97.43
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1.52 6.66 20.53 88886262 273904650
sda1 0.00 0.00 0.00 2484 18
sda2 1.46 5.68 18.95 75741946 252831120
sda3 0.06 0.98 1.58 13141400 21073512
# data here
sdb 103.06 13964.72 2718.28 186315436859 36266884800
sdb1 103.06 13964.72 2718.28 186315436235 36266884800
# commit logs here
sdc 1.47 1.71 309.36 22800725 4127423000
sdc1 1.47 1.71 309.36 22799901 4127423000
We're running on a beefy 64-bit Nehalem, so mmap should be
available/possible.
I need to check with our Cassandra lead when he's available as to why we're
not using mmap or auto.
>From /opt/cassandra/conf/storage-conf.xml.
<DiskAccessMode>standard</DiskAccessMode>
Heap size is 16gb.
JVM_OPTS=" \
-ea \
-Xms16G \
-Xmx16G \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+CMSParallelRemarkEnabled \
-XX:SurvivorRatio=8 \
-XX:MaxTenuringThreshold=1 \
-XX:CMSInitiatingOccupancyFraction=75 \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:+UseCompressedOops \
-XX:+UseThreadPriorities \
-XX:ThreadPriorityPolicy=42 \
-Dcassandra.compaction.priority=1"
If I've omitted any key infos, please advise and I'll provide.
Thanks,
David
