Okay, just thought I’d throw it out there. I just so happened to miss that
little tidbit from the upgrade guide and got bit by it.
That is an odd problem, then. I hope you find your answer. Please update
us if you figure it out. Very curious to hear the answer.
On Sat, Oct 13, 2018 at 4:09 PM Naik, Ninad <ninad.n...@epsilon.com> wrote:
> Thanks James. Yeah, we're using the datastax java driver. But we're on
> version 2.1.10.2. And we are not using the client side timestamps.
> ------------------------------
> *From:* James Carman <ja...@carmanconsulting.com>
> *Sent:* Saturday, October 13, 2018 12:55:51 PM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
> Are you using the Java driver? The default timestamp generator is
> client-side now (since 3.0 I believe), so if you don’t change it, you need
> your client clocks to be in sync also.
> On Sat, Oct 13, 2018 at 1:24 PM Naik, Ninad <ninad.n...@epsilon.com>
> wrote:
>
> Thanks Maitrayee. I should have mentioned this as one of the things we
> verified. The clocks on cassandra nodes are in sync.
> ------------------------------
> *From:* maitrayee shah <koolja...@yahoo.com.INVALID>
> *Sent:* Friday, October 12, 2018 6:40:25 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
> We have seen inconsistent read if the clock on the nodes are not in sync.
>
>
> Thank you
>
> Sent from my iPhone
>
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad <ninad.n...@epsilon.com> wrote:
>
> Hello,
>
> We're seeing inconsistent data while doing reads on cassandra. Here are
> the details:
>
> It's is a wide column table. The columns can be added my multiple
> machines, and read by multiple machines. The time between writes and reads
> are in minutes, but sometimes can be in seconds. Writes happen every 2
> minutes.
>
> Now, while reading we're seeing the following cases of inconsistent reads:
>
> - One column was added. If a read was done after the column was added
> (20 secs to 2 minutes after the write), Cassandra returns no data. As if
> the key doesn't exist. If the application retries, it gets the data.
> - A few columns exist for a row key. And a new column 'n' was added.
> Again, a read happens a few minutes after the write. This time, only the
> latest column 'n' is returned. In this case the app doesn't know that the
> data is incomplete so it doesn't retry. If we manually retry, we see all
> the columns.
> - A few columns exist for a row key. And a new column 'n' is added.
> When a read happens after the write, all columns but 'n' are returned.
>
> Here's what we've verified:
>
> - Both writes and reads are using 'LOCAL_QUORUM' consistency level.
> - The replication is within local data center. No remote data center
> is involved in the read or write.
> - During the inconsistent reads, none of the nodes are undergoing GC
> pauses
> - There are no errors in cassandra logs
> - Reads always happen after the writes.
>
> A few other details: Cassandra version: 2.1.9 DataStax java driver
> version: 2.1.10.2 Replication Factor: 3
>
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
>
> Here's how the table definition looks like:
>
> CREATE TABLE "MY_TABLE" (
> key text,
> sub_key text,
> value text,
> PRIMARY KEY ((key), sub_key)
> ) WITH
> bloom_filter_fp_chance=0.010000 AND
> caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
> comment='' AND
> dclocal_read_repair_chance=0.100000 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.000000 AND
> default_time_to_live=0 AND
> speculative_retry='ALWAYS' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
>
> Please point us in the right direction. Thanks !
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
