Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Rob Coli
On Thu, Sep 27, 2012 at 3:25 PM, Arya Goudarzi wrote: > rcoli helped me investigate this issue. The mystery was that the segment of > commit log was probably not fsynced to disk since the setting was set to > periodic with 10 second delay and CRC32 checksum validation failed skipping > the reply,

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Arya Goudarzi
rcoli helped me investigate this issue. The mystery was that the segment of commit log was probably not fsynced to disk since the setting was set to periodic with 10 second delay and CRC32 checksum validation failed skipping the reply, so what happened in my scenario can be explained by this. I am

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Arya Goudarzi
I was restarting Cassandra nodes again today. 1 hour later my support team let me know that a customer has reported some missing data. I suppose this is the same issue. The application logs show that our client got success from the Thrift log and proceeded with responding to the user and I could gr

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Arya Goudarzi
Thanks for your reply. I did grep on the commit logs for the offending key and grep showed Binary file matches. I am trying to use this tool to extract the commitlog and actually confirm if the mutation was a write: https://github.com/carloscm/cassandra-commitlog-extract.git On Thu, Sep 27, 2012

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-27 Thread Sylvain Lebresne
> I can verify the existence of the key that was inserted in Commitlogs of both > replicas however it seams that this record was never inserted. Out of curiosity, how can you verify that? -- Sylvain

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-26 Thread Arya Goudarzi
(time to live ) set on the columns... >> >> -- >> Date: Tue, 25 Sep 2012 19:56:19 -0700 >> Subject: 1.1.5 Missing Insert! Strange Problem >> From: gouda...@gmail.com >> To: user@cassandra.apache.org >> >> >> Hi

Re: 1.1.5 Missing Insert! Strange Problem

2012-09-26 Thread Arya Goudarzi
No. We don't use TTLs. On Tue, Sep 25, 2012 at 11:47 PM, Roshni Rajagopal < roshni_rajago...@hotmail.com> wrote: > By any chance is a TTL (time to live ) set on the columns... > > -- > Date: Tue, 25 Sep 2012 19:56:19 -0700 > Subject: 1.1

RE: 1.1.5 Missing Insert! Strange Problem

2012-09-25 Thread Roshni Rajagopal
By any chance is a TTL (time to live ) set on the columns... Date: Tue, 25 Sep 2012 19:56:19 -0700 Subject: 1.1.5 Missing Insert! Strange Problem From: gouda...@gmail.com To: user@cassandra.apache.org Hi All, I have a 4 node cluster setup in 2 zones with NetworkTopology strategy and strategy

1.1.5 Missing Insert! Strange Problem

2012-09-25 Thread Arya Goudarzi
Hi All, I have a 4 node cluster setup in 2 zones with NetworkTopology strategy and strategy options for writing a copy to each zone, so the effective load on each machine is 50%. Symptom: I have a column family that has gc grace seconds of 10 days (the default). On 17th there was an insert done t