Hello Folks,
I've an virtualized environment running with VMware where Cassandra is humming
on NFS mounted storage. As the application load increases ,they increase number
of nodes in data center however writes are getting slower, nodes are flapping
and application complains in write performance. I can see compaction is getting
behind, flush writers are struggling - symptoms that points to storage system
but system Admins says they have expensive, topnotch SSDs under data volume.
I know having nfs is not recommended though getting totally new h/w is not
possible at this time. I'm wondering if there is better way to utilize
available SAN resources to run with Cassandra ? I'll much appreciate your
insight here.
Thanks/Asad
Nfsiostat
=======================
$ nfsiostat /cassandra/data
172.XX.XX.16:/vol/cns_mount_0020_data mounted on /cassandra/data:
op/s rpc bklog
839.33 0.00
read: ops/s kB/s kB/op retrans
avg RTT (ms) avg exe (ms)
236.611 13471.729 56.936 20 (0.0%)
6.125 13.236
write: ops/s kB/s kB/op retrans
avg RTT (ms) avg exe (ms)
302.627 19048.744 62.945 3 (0.0%)
23.154 253.329
nodetool tpstats
==============
$ nodetool tpstats
Pool Name Active Pending Completed Blocked
All time blocked
ReadStage 0 0 3474362 0
0
MiscStage 0 0 0 0
0
CompactionExecutor 12 337 47130 0
0
MutationStage 256 238996 313857751 0
0
MemtableReclaimMemory 0 0 27081 0
0
PendingRangeCalculator 0 0 37 0
0
GossipStage 0 0 173852 0
0
SecondaryIndexManagement 0 0 0 0
0
HintsDispatcher 0 0 0 0
0
RequestResponseStage 0 0 337132828 0
0
Native-Transport-Requests 1 0 120206509 0
7073177
ReadRepairStage 0 0 8934 0
0
CounterMutationStage 0 0 0 0
0
MigrationStage 1 2 102 0
0
MemtablePostFlush 1 223 27753 0
0
PerDiskMemtableFlushWriter_0 2 2 27095 0
0
ValidationExecutor 0 0 0 0
0
Sampler 0 0 0 0
0
MemtableFlushWriter 5 175 27093 0
0
InternalResponseStage 0 0 46 0
0
ViewMutationStage 0 0 0 0
0
AntiEntropyStage 0 0 0 0
0
CacheCleanupExecutor 0 0 0 0
0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
HINT 944297
MUTATION 15707984
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 3195
