[
https://issues.apache.org/jira/browse/CASSANDRA-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-5665:
-----------------------------------
Attachment: 5665-v1.diff
The attached patch modifies Gossiper.handleMajorStateChanged by checking if the
the endpoint already exists in the endpointStateMap, and adds any previous
ApplicationState fields to the new epState
if a) the AppState does not exist in the new epState struct or b) has AppState
whose version is greater than that in the epState.
One the surface the patch is straight forward, but I'm not sure if there's some
subtle bugs that might creep in with retaining previous state (although that
state might get replaced anyways in a very short time). Thus, while the patch
fixes 'my problem', I'm not sure if this is the safest way to resolve.
> Gossiper.handleMajorStateChange can lose existing node ApplicationState
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-5665
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5665
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.5
> Reporter: Jason Brown
> Priority: Minor
> Labels: gossip, upgrade
> Fix For: 1.2.6, 2.0 beta 1
>
> Attachments: 5665-v1.diff
>
>
> Dovetailing on #5660, I discovered that further along during an upgrade, when
> more nodes are on the new major version, a node the previous version can get
> passed some incomplete Gossip info about another, already upgraded node, and
> the older node drops AppStat info about that node.
> I think what happens is that a 1.1 node (older rev) gets gossip info from a
> 1.2 node (A), which includes incomplete (lacking some AppState data) gossip
> info about another 1.2 node (B). The 1.1 node, which has marked incorrectly
> kicked node B out of gossip due to the bug described in #5660, then takes
> that incomplete node B info and wholesale replaces any previous known state
> about node B in Gossiper.handleMajorStateChanged. Thus, if we previously had
> DC/RACK info, it'll get dropped as part of the
> endpointStateMap.put(endpointstate). When the data being pased is incomplete,
> 1.1 will start referencing node B and gets into the NPE situation in #5498.
> Anecdotally, this bad state is short-lived, less than a few minutes, even as
> short as ten seconds, until gossip catches up and properly propagates the
> AppState data. Furthermore, when upgrading a two datacenter, 48 node cluster,
> it only occurred on two nodes for less than a minute each. Thus, the scope
> seems limited but can occur.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
