If anyone has seen this issue and knows a fix, it would be a great help!
Thanks in advance.
Jaydeep
On Fri, Sep 2, 2022 at 1:56 PM Jaydeep Chovatia <chovatia.jayd...@gmail.com>
wrote:
> Hi,
>
> We are running a production Cassandra version (3.0.14) with 256 tokens
> v-node configuration. Occasionally, we see that different nodes show
> different ownership for the same key. Only a node restart corrects;
> otherwise, it continues to behave in a split-brain.
>
> Say, for example,
>
> *NodeA*
> nodetool getendpoints ks1 table1 10
> - n1
> - n2
> - n3
>
> *NodeB*
> nodetool getendpoints ks1 table1 10
> - n1
> - n2
> *- n5*
>
> If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The
> majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a
> few show this issue, and restarting them solves the problem.
>
> To me, it seems I think Cassandra's Gossip cache and StorageService cache
> (TokenMetadata) are having some sort of cache coherence.
>
> Anyone has observed this behavior?
> Any help would be highly appreciated.
>
> Jaydeep
>
