What have you set these to?
# commitlog_sync may be either "periodic" or "batch."
# When in batch mode, Cassandra won't ack writes until the commit log
# has been fsynced to disk. It will wait up to
# commitlog_sync_batch_window_in_ms milliseconds for other writes, before
# performing the sync.
#
# commitlog_sync: batch
# commitlog_sync_batch_window_in_ms: 50
#
# the other option is "periodic" where writes may be acked immediately
# and the CommitLog is simply synced every commitlog_sync_period_in_ms
# milliseconds.
commitlog_sync: periodic
commitlog_sync_period_in_ms: 1000
On Tue, Sep 10, 2013 at 10:42 AM, Nate McCall <n...@thelastpickle.com>wrote:
> With SSDs, you can turn up memtable_flush_writers - try 3 initially (1 by
> default) and see what happens. However, given that there are no entries in
> 'All time blocked' for such, they may be something else.
>
> How are you inserting the data?
>
>
> On Tue, Sep 10, 2013 at 12:40 PM, Keith Freeman <8fo...@gmail.com> wrote:
>
>>
>> On 09/10/2013 11:17 AM, Robert Coli wrote:
>>
>> On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman <8fo...@gmail.com> wrote:
>>
>>> On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog
>>> and data
>>
>>
>> On SSD, you don't need to separate commitlog and data. You only win
>> from this separation if you have a head to not-move between appends to the
>> commit log. You will get better IO from a strip with an additional SSD.
>>
>> Right, actually both partitions are on the same SSD. Assuming you meant
>> "stripe", would that really make a difference
>>
>>
>>
>>> Pool Name Active Pending Completed Blocked
>>>> All time blocked
>>>> MutationStage 1 9 290394 0
>>>> 0
>>>> FlushWriter 1 2 20 0
>>>> 0
>>>>
>>>
>>
>>> I can't seem find information about the real meaning of MutationStage,
>>> is this just normal for lots of inserts?
>>>
>>
>> The mutation stage is the stage in which mutations to rows in memtables
>> ("writes") occur.
>>
>> The FlushWriter stage is the stage that turns memtables into SSTables
>> by flushing them.
>>
>> However, 9 pending mutations is a very small number. For reference on
>> an overloaded cluster which was being written to death I recently saw....
>> 1216434 pending MutationStage. What problem other than "high CPU load" are
>> you experiencing? 2 Pending FlushWriters is slightly suggestive of some
>> sort of bound related to flushing..
>>
>> So the basic problem is that write performance is lower than I expected.
>> I can't get sustained writing of 5000 ~1024-byte records / sec at RF=2 on a
>> good 3-node cluster, and my only guess is that's because of the heavy CPU
>> loads on the server (loads over 10 on 4-CPU systems). I've tried both a
>> single client writing 5000 rows/second and 2 clients (on separate boxes)
>> writing 2500 rows/second, and in both cases the server(s) doesn't respond
>> quickly enough to maintain that rate. It keeps up ok with 2000 or 3000
>> rows per second (and has lower server loads).
>>
>>
>>
>
