How much of the data you are writing is going against the same row key? I've experienced some issues using CQL to write a full wide-row at once (across multiple threads) that exhibited some of the symptoms you have described (i.e., high cpu, dropped mutations).
This question goes into it a bit more: http://stackoverflow.com/questions/18522191/using-cassandra-and-cql3-how-do-you-insert-an-entire-wide-row-in-a-single-reque . I was able to solve my issue by switching to using the thrift batch_mutate to write a full wide-row at once instead of using many CQL INSERT statements. -Paul > -----Original Message----- > From: Keith Freeman [mailto:8fo...@gmail.com] > Sent: Wednesday, September 11, 2013 9:16 AM > To: user@cassandra.apache.org > Subject: Re: heavy insert load overloads CPUs, with MutationStage pending > > > On 09/10/2013 11:42 AM, Nate McCall wrote: > > With SSDs, you can turn up memtable_flush_writers - try 3 initially (1 > > by default) and see what happens. However, given that there are no > > entries in 'All time blocked' for such, they may be something else. > Tried that, it seems to have reduced the loads a little after everything > warmed-up, but not much. > > > > How are you inserting the data? > > A java client on a separate box using the datastax java driver, 48 threads > writing 100 records each iteration as prepared batch statements. > > At 5000 records/sec, the servers just can't keep up, so the client backs up. > That's only 5M of data/sec, which doesn't seem like much. As I mentioned, > switching to SSDs didn't help much, so I'm assuming at this point that the > server overloads are what's holding up the client.
