Hi Mick, Thanks for replying!
I did not reset repairedAt and ran repair with -pr directly. That’s probably why the inconsistency occurred. As our tables are pretty big, full repair takes many days to finish. Given the 10 days gc period, it means repair almost will run all the time. Consistency is more important to us and the cluster takes the same amount of write and read requests. Temporary outage is allowed but if a dead node can’t come back in time, we will go back to Quorum mode. What’s new in 3.11.3? We’ve been running on C* for almost 2 years. The biggest pain point is about repair. Especially with 3.11.1, incremental repair doesn’t work well compared to our experience with 3.10. Maybe it’s just because our data size wasn’t that big before upgrade... Thanks, Li > On Jul 11, 2018, at 05:32, Mick Semb Wever <m...@thelastpickle.com> wrote: > > > Li, > > >> I’ve confirmed that the inconsistency issues disappeared after repair >> finished. >> >> Anything changed with repair in 3.11.1? One difference I noticed is that the >> validation step during repair could turn down the node upon large tables, >> which never happen in 3.10. I had to throttle validation requests to let it >> pass. Also I switched back to -pr instead of incremental repair which is a >> resource killer and often hangs for the first node to be repaired. > > > When you switched back to non-incremental did you set `repairedAt` on all > sstables (on all nodes) back to zero (or unrepaired state)? > This should have been done with `sstablerepairedset --is-unrepaired … ` while > the node is stopped. > > >> To address the inconsistency issue, I could do Write All and Read One by >> giving up availability and stop running repair. Any comments on that? > > > You loose availability doing this, and at the number of reads you're doing I > would not recommend it. > You could think about using a fallback strategy that initially tries CL.ALL > and falls back to CL.QUORUM. But this is a hack, could overload your cluster, > and if there's any correlation to dropped messages or flapping nodes won't > help. > > I'd also be prepared to upgrade to 3.11.3, when it does get released. > > regards, > Mick > > -- > Mick Semb Wever > Australia > > The Last Pickle > Apache Cassandra Consulting > http://www.thelastpickle.com
