The commitlog_total_space_in_mb was not set, I set it to avoid having the same problem in the future.
I am aware of the over-counting problem introduced by the counters. The point is that I use them to make statistics per hours. I can understand having some wrong counts in the column corresponding to the crash time, but how to explain that all my counts since the start (months ago) have become wrong after the crash ? After the crash I tried to repair my entire keyspace from one of the 2 nodes and this made my server crash again, no idea why. Can this failed repair be at the origin of the corrupted data ? I'm still replaying all my counts of the past months and I'm afraid this kind of bug could happen again... I was using cassandra for months without any issue. Alain 2012/4/11 aaron morton <aa...@thelastpickle.com> > However after recovering from this issue (freeing some space and fixing >> the value of "commitlog_total_space_in_mb" in cassandra.yaml) >> > Did the commit log grow larger than commitlog_total_space_in_mb ? > > I realized that all statistics were all destroyed. I have bad values on >> every single counter since I start using them (september) ! >> > Counter operations are not idempotent. If you client retries a counter > operation it may result in the increment been applied twice. Could this > have been your issue ? > > Cheers > > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 11/04/2012, at 2:35 AM, Alain RODRIGUEZ wrote: > > By the way, I am using Cassandra 1.0.7, CL = ONE (R/W), RF = 2, 2 EC2 > c1.medium nodes cluster > > Alain > > 2012/4/10 Alain RODRIGUEZ <arodr...@gmail.com> > >> Hi, I'm experimenting a strange and very annoying phenomena. >> >> I had a problem with the commit log size which grew too much and full one >> of the hard disks in all my nodes almost at the same time (2 nodes only, >> RF=2, so the 2 nodes are behaving exactly in the same way) >> >> My data are mounted in an other partition that was not full. However >> after recovering from this issue (freeing some space and fixing the value >> of "commitlog_total_space_in_mb" in cassandra.yaml) I realized that all >> statistics were all destroyed. I have bad values on every single counter >> since I start using them (september) ! >> >> Does anyone experimented something similar or have any clue on this ? >> >> Do you need more information ? >> >> Alain >> > > >
