Ok, your results are pretty impressive, I'm giving it a try. I've made
some initial attempts to use Astyanax 1.56.37, but have some troubles:
- it's not compatible with 1.2.8 client-side ( NoSuchMethodError's on
org.apache.cassandra.thrift.TBinaryProtocol, which changed it's
signature since 1.2.5)
- even switching to C* 1.2.5 servers, it's been difficult getting
simple examples to work unless I use CF's that have "WITH COMPACT STORAGE"
How did you handle these problems? How much effort did it take you to
switch from datastax to astyanax?
I feel like I'm getting lost in a pretty deep rabbit-hole here.
On 09/11/2013 03:03 PM, Paul Cichonski wrote:
I was reluctant to use the thrift as well, and I spent about a week trying to
get the CQL inserts to work by partitioning the INSERTS in different ways and
tuning the cluster.
However, nothing worked remotely as well as the batch_mutate when it came to
writing a full wide-row at once. I think Cassandra 2.0 makes CQL work better
for these cases (CASSANDRA-4693), but I haven't tested it yet.
-Paul
-----Original Message-----
From: Keith Freeman [mailto:8fo...@gmail.com]
Sent: Wednesday, September 11, 2013 1:06 PM
To: user@cassandra.apache.org
Subject: Re: heavy insert load overloads CPUs, with MutationStage pending
Thanks, I had seen your stackoverflow post. I've got hundreds of
(wide-) rows, and the writes are pretty well distributed across them.
I'm very reluctant to drop back to the thrift interface.
On 09/11/2013 10:46 AM, Paul Cichonski wrote:
How much of the data you are writing is going against the same row key?
I've experienced some issues using CQL to write a full wide-row at once
(across multiple threads) that exhibited some of the symptoms you have
described (i.e., high cpu, dropped mutations).
This question goes into it a bit
more:http://stackoverflow.com/questions/18522191/using-cassandra-and-
cql3-how-do-you-insert-an-entire-wide-row-in-a-single-reque . I was able to
solve my issue by switching to using the thrift batch_mutate to write a full
wide-row at once instead of using many CQL INSERT statements.
-Paul
-----Original Message-----
From: Keith Freeman [mailto:8fo...@gmail.com]
Sent: Wednesday, September 11, 2013 9:16 AM
To:user@cassandra.apache.org
Subject: Re: heavy insert load overloads CPUs, with MutationStage
pending
On 09/10/2013 11:42 AM, Nate McCall wrote:
With SSDs, you can turn up memtable_flush_writers - try 3 initially
(1 by default) and see what happens. However, given that there are
no entries in 'All time blocked' for such, they may be something else.
Tried that, it seems to have reduced the loads a little after
everything warmed-up, but not much.
How are you inserting the data?
A java client on a separate box using the datastax java driver, 48
threads writing 100 records each iteration as prepared batch statements.
At 5000 records/sec, the servers just can't keep up, so the client backs up.
That's only 5M of data/sec, which doesn't seem like much. As I
mentioned, switching to SSDs didn't help much, so I'm assuming at
this point that the server overloads are what's holding up the client.
