Re: Inconsistent Quorum Read after Quorum Write

2018-07-11 Thread Mick Semb Wever
Li, I did not reset repairedAt and ran repair with -pr directly. That’s > probably why the inconsistency occurred. > Yes, this will be a likely cause. There's enough docs out there to help you with this. Shout out if not. > As our tables are pretty big, full repair takes many days to finish. G

Re: Inconsistent Quorum Read after Quorum Write

2018-07-11 Thread Visa
Hi Mick, Thanks for replying! I did not reset repairedAt and ran repair with -pr directly. That’s probably why the inconsistency occurred. As our tables are pretty big, full repair takes many days to finish. Given the 10 days gc period, it means repair almost will run all the time. Consistency

Re: Inconsistent Quorum Read after Quorum Write

2018-07-11 Thread Mick Semb Wever
Li, I’ve confirmed that the inconsistency issues disappeared after repair > finished. > > Anything changed with repair in 3.11.1? One difference I noticed is that > the validation step during repair could turn down the node upon large > tables, which never happen in 3.10. I had to throttle valida

Re: Inconsistent Quorum Read after Quorum Write

2018-07-04 Thread Visa
I’ve confirmed that the inconsistency issues disappeared after repair finished. Anything changed with repair in 3.11.1? One difference I noticed is that the validation step during repair could turn down the node upon large tables, which never happen in 3.10. I had to throttle validation requests

Re: Inconsistent Quorum Read after Quorum Write

2018-07-03 Thread Visa
Thanks for replying! The version is C* 3.11.1. The quorum write and read are done in java code (spark streaming) and in async mode within the same session. Could not reproduce it via cqlsh yet. With the same session, execute the async write, and in the callback execute the async read. To be

Re: Inconsistent Quorum Read after Quorum Write

2018-07-03 Thread kurt greaves
Shouldn't happen. Any chance you could trace the queries, or have you been able to reproduce it? Also, what version of Cassandra? On Wed., 4 Jul. 2018, 06:41 Visa, wrote: > Hi all, > > We recently experienced an unexpected behavior with C* consistency. > > For example, a table t consists of 4 co

Inconsistent Quorum Read after Quorum Write

2018-07-03 Thread Visa
Hi all, We recently experienced an unexpected behavior with C* consistency. For example, a table t consists of 4 columns - pk , a, b and c. We perform Quorum write and then Quorum read (RF=3 / LCS compaction). The consistency seems to break while repairing is running(repair -pr). Say, a record