Re: "MessageDeserializationTask.java (line 47) dropping message" errors

2010-08-20 Thread Ronald Park
Oh, excellent. Ok, my code does some retrying (including some sleep then retry logic) when I get a TimedOut, so I think I'm actually just fine. Whew. :) I had just noticed the dropped message in Cassandra's log today but had seen the time outs in my log while it was running. I just didn't conne

Re: "MessageDeserializationTask.java (line 47) dropping message" errors

2010-08-20 Thread Jonathan Ellis
On Fri, Aug 20, 2010 at 5:02 PM, Ronald Park wrote: > Well this is bad news... Why is this logged only as a WARN?  Seems quite > SEVERE to me. That's as high as log4j goes, without going all the way to ERROR which we reserve for bugs. (High load is not a bug.) > Fortunately for my app, I can go

Re: "MessageDeserializationTask.java (line 47) dropping message" errors

2010-08-20 Thread Ronald Park
Well this is bad news... Why is this logged only as a WARN? Seems quite SEVERE to me. Fortunately for my app, I can go back to my source, recreate the data and reload it at a slower rate (and cross my fingers that this rate is 'slow enough')... unfortunately, it took me 20 hours to load the data

Re: "MessageDeserializationTask.java (line 47) dropping message" errors

2010-08-20 Thread Jonathan Ellis
updates being dropped on the floor is exactly what that message means (the batch_mutate you send are decomposed under the hood to individual rows for replication, which is why the number is greater than your 3000) On Fri, Aug 20, 2010 at 3:50 PM, Ronald Park wrote: > How about this message: > >

Re: "MessageDeserializationTask.java (line 47) dropping message" errors

2010-08-20 Thread Ronald Park
How about this message: WARN [DroppedMessagesLogger] 2010-08-20 08:02:27,668 MessagingService.java (lin e 512) Dropped 46469 messages in the last 1000ms For this application, I also have a 3-node cluster but I'm running a version of Cassandra built off the trunk (we wanted the 'time to live' fea

Re: "MessageDeserializationTask.java (line 47) dropping message" errors

2010-08-18 Thread Jonathan Ellis
This is Cassandra trying to protect itself from the load spike. You probably need to balance your nodes, tune for performance, add capacity, or some combination. On Wed, Aug 18, 2010 at 6:16 AM, Jianing Hu wrote: > We have a 3-node cluster of Cassandra 0.6.4, on one of them there's a > ton of er

"MessageDeserializationTask.java (line 47) dropping message" errors

2010-08-18 Thread Jianing Hu
We have a 3-node cluster of Cassandra 0.6.4, on one of them there's a ton of error message like the following one: WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-18 04:10:57,767 MessageDeserializationTask.java (line 47) dropping message (1ms past timeout) The nod's load spikes and is super slow to rea