With SSDs, you can turn up memtable_flush_writers - try 3 initially (1 by
default) and see what happens. However, given that there are no entries in
'All time blocked' for such, they may be something else.
How are you inserting the data?
On Tue, Sep 10, 2013 at 12:40 PM, Keith Freeman <8fo...@gmail.com> wrote:
>
> On 09/10/2013 11:17 AM, Robert Coli wrote:
>
> On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman <8fo...@gmail.com> wrote:
>
>> On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog
>> and data
>
>
> On SSD, you don't need to separate commitlog and data. You only win from
> this separation if you have a head to not-move between appends to the
> commit log. You will get better IO from a strip with an additional SSD.
>
> Right, actually both partitions are on the same SSD. Assuming you meant
> "stripe", would that really make a difference
>
>
>
>> Pool Name Active Pending Completed Blocked
>>> All time blocked
>>> MutationStage 1 9 290394 0
>>> 0
>>> FlushWriter 1 2 20 0
>>> 0
>>>
>>
>
>> I can't seem find information about the real meaning of MutationStage,
>> is this just normal for lots of inserts?
>>
>
> The mutation stage is the stage in which mutations to rows in memtables
> ("writes") occur.
>
> The FlushWriter stage is the stage that turns memtables into SSTables by
> flushing them.
>
> However, 9 pending mutations is a very small number. For reference on an
> overloaded cluster which was being written to death I recently saw....
> 1216434 pending MutationStage. What problem other than "high CPU load" are
> you experiencing? 2 Pending FlushWriters is slightly suggestive of some
> sort of bound related to flushing..
>
> So the basic problem is that write performance is lower than I expected.
> I can't get sustained writing of 5000 ~1024-byte records / sec at RF=2 on a
> good 3-node cluster, and my only guess is that's because of the heavy CPU
> loads on the server (loads over 10 on 4-CPU systems). I've tried both a
> single client writing 5000 rows/second and 2 clients (on separate boxes)
> writing 2500 rows/second, and in both cases the server(s) doesn't respond
> quickly enough to maintain that rate. It keeps up ok with 2000 or 3000
> rows per second (and has lower server loads).
>
>
>
