Thanks Mick.
Yeah we are planning to try with tracing and by enabling trace level logs for a short duration. I will update this thread with the related details. One other thing we verified is that these partial reads happen all across the cluster. It's not limited to certain cassandra nodes. Meanwhile, can following scenarios (however unlikely they might seem) ever happen? - Can a bloom filter malfunction (Ex: For some reason, bloom filter says to skip a SSTable which has the key) - Can the co-ordinator ever send a read request to a wrong replica (Ex: for some reason it calculated a wrong decorated key)? - Can a replica ever drop a read in case of an exception? - Can read timeouts for other wide rows (we've a few rows with > 100k cols) impact the entire cluster to behave in a delayed manner? Thanks again for all of your help ! ________________________________ From: Mick Semb Wever <m...@thelastpickle.com> Sent: Wednesday, October 24, 2018 4:12:57 AM To: user@cassandra.apache.org Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) [ This email has been sent from a source external to Epsilon. Please use caution when clicking links or opening attachments. ] Ninad, > Here's a bit more information: > > -Few rows in this column family can grow quite wide (> 100K columns) > > -But we keep seeing this behavior most frequently with rows with just 1 or > two columns . The typical behavior is: Machine A adds a new row and a column. > 30-60 seconds later Machine B tries to read this row. It doesn't find the > row. So the application retries within 500ms. This time it finds the row. You wrote a lot of useful info in your original post, sorry I missed it in my first reply. Only thing there that stands out, apart from short reads that Jeff's already pointed out, is the use of `speculative_retry='ALWAYS'`. Has there topology changes in your cluster recently? Next step would be to try and repeat it with tracing. regards, Mick -- Mick Semb Wever Australia The Last Pickle Apache Cassandra Consulting http://www.thelastpickle.com --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this e-mail and delete the message and any attachments from your computer.
