On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog
and data, I get high CPU loads during a heavy-ish wide-row insert load
into a single CF (5000 1k inserts/sec), e.g. uptime load avg for last
minute 18/11/10. Checking tpstats, I see MutationStage pending on all
the nodes, e.g.:
Pool Name Active Pending Completed Blocked All
time blocked
ReadStage 0 0 144 0
0
RequestResponseStage 0 0 243529 0
0
MutationStage 1 9 290394 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 1014 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage 0 0 13 0
0
MemtablePostFlusher 1 2 35 0
0
FlushWriter 1 2 20 0
0
MiscStage 0 0 1 0
0
commitlog_archiver 0 0 0 0
0
I can't seem find information about the real meaning of MutationStage,
is this just normal for lots of inserts?
Also, switching from spinning disks to SSDs didn't seem to significantly
improve insert performance, so it seems clear my use-case it totally
CPU-bound. Cassandra docs say "Insert-heavy workloads are CPU-bound in
Cassandra before becoming memory-bound.", so I guess that's what I'm
seeing, but there's no explanation. So I'm wonder what's overloading my
CPUs, and is there anything I can do about it short of adding more nodes?
.
