Well this is bad news... Why is this logged only as a WARN? Seems quite SEVERE to me.
Fortunately for my app, I can go back to my source, recreate the data and reload it at a slower rate (and cross my fingers that this rate is 'slow enough')... unfortunately, it took me 20 hours to load the data (200M records) at a rate of 3K/sec. It looks like I only dropped about 600K records out of 200M (.3%) ... but no way to tell which ones were dropped. :( (I know, I know, more machines... they are on order... sigh.) Is there a way to configure Cassandra so that it won't do this? I'd rather have it run slow than drop anything on the floor. Thanks, Ron On Fri, Aug 20, 2010 at 4:58 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > updates being dropped on the floor is exactly what that message means > > (the batch_mutate you send are decomposed under the hood to individual > rows for replication, which is why the number is greater than your > 3000) > > On Fri, Aug 20, 2010 at 3:50 PM, Ronald Park > <ronald.p...@cbsinteractive.com> wrote: > > How about this message: > > > > WARN [DroppedMessagesLogger] 2010-08-20 08:02:27,668 > MessagingService.java > > (lin > > e 512) Dropped 46469 messages in the last 1000ms > > > > For this application, I also have a 3-node cluster but I'm running a > version > > of Cassandra built off the trunk (we wanted the 'time to live' feature > > before 0.7.beta was out). My application runs a daily loading script > which > > peaks at under 3000 batch_mutate requests per second, well below the 46K > in > > this log. Are these some sort of internal messages, like pings? Or are > > updates being dropped on the floor? > > > > Thanks, > > Ron > > > > On Wed, Aug 18, 2010 at 9:13 AM, Jonathan Ellis <jbel...@gmail.com> > wrote: > >> > >> This is Cassandra trying to protect itself from the load spike. You > >> probably need to balance your nodes, tune for performance, add > >> capacity, or some combination. > >> > >> On Wed, Aug 18, 2010 at 6:16 AM, Jianing Hu <jian...@gmail.com> wrote: > >> > We have a 3-node cluster of Cassandra 0.6.4, on one of them there's a > >> > ton of error message like the following one: > >> > WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-18 04:10:57,767 > >> > MessageDeserializationTask.java (line 47) dropping message (1ms past > >> > timeout) > >> > > >> > The nod's load spikes and is super slow to read from. Any ideas? > >> > > >> > Thanks. > >> > > >> > - Jianing > >> > > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of Riptano, the source for professional Cassandra support > >> http://riptano.com > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
