Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Rob Coli
On Thu, Sep 27, 2012 at 3:25 PM, Arya Goudarzi wrote: > rcoli helped me investigate this issue. The mystery was that the segment of > commit log was probably not fsynced to disk since the setting was set to > periodic with 10 second delay and CRC32 checksum validation failed skipping > the reply,

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Arya Goudarzi
rcoli helped me investigate this issue. The mystery was that the segment of commit log was probably not fsynced to disk since the setting was set to periodic with 10 second delay and CRC32 checksum validation failed skipping the reply, so what happened in my scenario can be explained by this. I am

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Arya Goudarzi
I was restarting Cassandra nodes again today. 1 hour later my support team let me know that a customer has reported some missing data. I suppose this is the same issue. The application logs show that our client got success from the Thrift log and proceeded with responding to the user and I could gr

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Arya Goudarzi
Thanks for your reply. I did grep on the commit logs for the offending key and grep showed Binary file matches. I am trying to use this tool to extract the commitlog and actually confirm if the mutation was a write: https://github.com/carloscm/cassandra-commitlog-extract.git On Thu, Sep 27, 2012

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Sylvain Lebresne
> I can verify the existence of the key that was inserted in Commitlogs of both > replicas however it seams that this record was never inserted. Out of curiosity, how can you verify that? -- Sylvain

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-26 Thread Arya Goudarzi
Any change anyone has seen the same mysterious issue? On Wed, Sep 26, 2012 at 12:03 AM, Arya Goudarzi wrote: > No. We don't use TTLs. > > > On Tue, Sep 25, 2012 at 11:47 PM, Roshni Rajagopal < > roshni_rajago...@hotmail.com> wrote: > >> By any chance is a TTL (time to live ) set on the columns.

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-26 Thread Arya Goudarzi
No. We don't use TTLs. On Tue, Sep 25, 2012 at 11:47 PM, Roshni Rajagopal < roshni_rajago...@hotmail.com> wrote: > By any chance is a TTL (time to live ) set on the columns... > > -- > Date: Tue, 25 Sep 2012 19:56:19 -0700 > Subject: 1.1.5 Missing Insert! Strange Prob

RE: 1.1.5 Missing Insert! Strange Problem

2012-09-25 Thread Roshni Rajagopal
By any chance is a TTL (time to live ) set on the columns... Date: Tue, 25 Sep 2012 19:56:19 -0700 Subject: 1.1.5 Missing Insert! Strange Problem From: gouda...@gmail.com To: user@cassandra.apache.org Hi All, I have a 4 node cluster setup in 2 zones with NetworkTopology strategy and strategy op