On 09/10/2013 11:17 AM, Robert Coli wrote:
On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman <8fo...@gmail.com
<mailto:8fo...@gmail.com>> wrote:
On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for
commitlog and data
On SSD, you don't need to separate commitlog and data. You only win
from this separation if you have a head to not-move between appends to
the commit log. You will get better IO from a strip with an additional
SSD.
Right, actually both partitions are on the same SSD. Assuming you
meant "stripe", would that really make a difference
Pool Name Active Pending Completed
Blocked All time blocked
MutationStage 1 9 290394 0
0
FlushWriter 1 2 20 0
0
I can't seem find information about the real meaning of
MutationStage, is this just normal for lots of inserts?
The mutation stage is the stage in which mutations to rows in
memtables ("writes") occur.
The FlushWriter stage is the stage that turns memtables into SSTables
by flushing them.
However, 9 pending mutations is a very small number. For reference on
an overloaded cluster which was being written to death I recently
saw.... 1216434 pending MutationStage. What problem other than "high
CPU load" are you experiencing? 2 Pending FlushWriters is slightly
suggestive of some sort of bound related to flushing..
So the basic problem is that write performance is lower than I
expected. I can't get sustained writing of 5000 ~1024-byte records /
sec at RF=2 on a good 3-node cluster, and my only guess is that's
because of the heavy CPU loads on the server (loads over 10 on 4-CPU
systems). I've tried both a single client writing 5000 rows/second and
2 clients (on separate boxes) writing 2500 rows/second, and in both
cases the server(s) doesn't respond quickly enough to maintain that
rate. It keeps up ok with 2000 or 3000 rows per second (and has lower
server loads).
