Appreciate the reply, Kurt. I sanitized it out of the traces, but all trace outputs listed the same node for all three queries (1 working, 2 not working). Read repair chance set to 0.0 as recommended when using TWCS.
I'll check tpstats - in this environment, load is not an issue, but network issues may be. On Mon, Mar 20, 2017 at 2:42 PM, kurt greaves <k...@instaclustr.com> wrote: > As secondary indexes are stored individually on each node what you're > suggesting sounds exactly like a consistency issue. the fact that you read > 0 cells on one query implies the node that got the query did not have any > data for the row. The reason you would sometimes see different behaviours > is likely because of read repairs. The fact that the repair guides the > issue pretty much guarantees it's a consistency issue. > > You should check for dropped mutations in tpstats/logs and if they are > occurring try and stop that from happening (probably load related). You > could also try performing reads and writes at LOCAL_QUORUM for stronger > consistency, however note this has a performance/latency impact. > > >
