[
https://issues.apache.org/jira/browse/CASSANDRA-20659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17952840#comment-17952840
]
Sam Tunnicliffe commented on CASSANDRA-20659:
---------------------------------------------
Trunk should not be affected in the same way as earlier versions because:
* Node status is derived from ClusterMetadata not gossip, so although we still
propagate app states like STATUS_WITH_PORT for instance, these don't materially
affect things like replica placement etc. {{StorageService::onChange}} does
very little in trunk and could probably do even less.
* Paxos no longer relies on/mutates gossip state during electorate
verification.
Gossip state is now the source of truth only for transient node state like LOAD
& RPC_READY. For persistent state such as STATUS_WITH_PORT, DC, RACK, this is
all derived from ClusterMetadata.
So there is probably some scope on trunk for further cleaning up of the
processing of gossip state/changes in state, but I think this is mostly just
housekeeping.
> Gossip doesn't converge due to race condition when updating EndpointStates
> multiple fields
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-20659
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20659
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip
> Reporter: David Capwell
> Assignee: David Capwell
> Priority: Normal
> Fix For: 4.1.x, 5.0.x, 5.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The issue seen is during shrinks or token moves the cluster gets into a state
> where some of the nodes never converge and see the latest STATUS state for
> the changed peers.
> In testing this it was found that:
> 1) org.apache.cassandra.gms.Gossiper#applyStateLocally expects to run in a
> single thread, so doesn't take any locks
> 2) org.apache.cassandra.gms.Gossiper.GossipTask runs in another thread and
> uses a taskLock to avoid sending partial state
> 3) org.apache.cassandra.gms.Gossiper#applyNewStates gets called when the
> generation matches, and tries to apply the state sequentially.
> The theory (and test) is
> 1) localState.setHeartBeatState(remoteState.getHeartBeatState()); runs
> 2) something (gossip or paxos) read the state
> 3) localState.addApplicationStates(updatedStates); updates the state
> the "something" in step 2 sends around the heartbeat which cause others to
> see a higher max version, so the delta logic won't see the mutations done in
> step 3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
