Thanks Scott. I will prioritize upgrading to 3.0.27 and will circle back if
this issue persists.
Jaydeep
On Tue, Sep 6, 2022 at 3:45 PM C. Scott Andreas <sc...@paradoxica.net>
wrote:
> Hi Jaydeep,
>
> Thanks for reaching out and for bumping this thread.
>
> This is probably not the answer you’re after, but mentioning as it may
> address the issue.
>
> C* 3.0.14 was released over five years ago, with many hundreds of
> important bug fixes landing since July 2017. These include fixes for issues
> that have affected gossip in the past which may be related to this issue.
> Note that 3.0.14 also is susceptible to several critical data loss bugs
> including C-14513 and C-14515.
>
> I’d strongly recommend upgrading to Cassandra 3.0.27 as a starting point.
> If this doesn’t resolve your issue, members of the community may be in a
> better position to help triage a bug report against a current release of
> the database.
>
> - Scott
>
> On Sep 6, 2022, at 5:13 PM, Jaydeep Chovatia <chovatia.jayd...@gmail.com>
> wrote:
>
>
> If anyone has seen this issue and knows a fix, it would be a great help!
> Thanks in advance.
>
> Jaydeep
>
> On Fri, Sep 2, 2022 at 1:56 PM Jaydeep Chovatia <
> chovatia.jayd...@gmail.com> wrote:
>
>> Hi,
>>
>> We are running a production Cassandra version (3.0.14) with 256 tokens
>> v-node configuration. Occasionally, we see that different nodes show
>> different ownership for the same key. Only a node restart corrects;
>> otherwise, it continues to behave in a split-brain.
>>
>> Say, for example,
>>
>> *NodeA*
>> nodetool getendpoints ks1 table1 10
>> - n1
>> - n2
>> - n3
>>
>> *NodeB*
>> nodetool getendpoints ks1 table1 10
>> - n1
>> - n2
>> *- n5*
>>
>> If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The
>> majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a
>> few show this issue, and restarting them solves the problem.
>>
>> To me, it seems I think Cassandra's Gossip cache and StorageService cache
>> (TokenMetadata) are having some sort of cache coherence.
>>
>> Anyone has observed this behavior?
>> Any help would be highly appreciated.
>>
>> Jaydeep
>>
>
