[
https://issues.apache.org/jira/browse/CASSANDRA-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096182#comment-18096182
]
Stefan Miklosovic edited comment on CASSANDRA-21249 at 7/14/26 1:48 PM:
------------------------------------------------------------------------
[~curlylrt] thanks, from 4.0 CI (1) I see that
{{org.apache.cassandra.distributed.test.hostreplacement.AssassinateGracefullNodeTest}}
fails.
{code}
junit.framework.AssertionFailedError: Node node2 did not match state LEFT
{/127.0.0.2={generation=1784034306, RPC_READY=61:false, heartbeat=2147483647,
LOAD=28:26449.0, NET_VERSION=1:12, TOKENS=22:<hidden>,
INTERNAL_ADDRESS_AND_PORT=8:127.0.0.2:7012, RACK=14:rack0,
STATUS=60:shutdown,true, SCHEMA=43:e8bb1f8c-f621-3f8b-aa57-924a5a7dc5b4,
STATUS_WITH_PORT=59:shutdown,true, NATIVE_ADDRESS_AND_PORT=3:127.0.0.2:9042,
RELEASE_VERSION=5:4.0.21-SNAPSHOT,
HOST_ID=2:3ceff7f2-32fb-4167-9820-5d4b63ee72c2, SSTABLE_VERSIONS=6:big-nb,
DC=12:datacenter0}, /127.0.0.1={generation=1784034293,
INTERNAL_IP=10:127.0.0.1, heartbeat=188, LOAD=164:96368.0, NET_VERSION=1:12,
TOKENS=22:<hidden>, RPC_ADDRESS=4:127.0.0.1,
INTERNAL_ADDRESS_AND_PORT=8:127.0.0.1:7012, RACK=14:rack0,
STATUS=24:NORMAL,-3074457345618258603,
SCHEMA=58:e8bb1f8c-f621-3f8b-aa57-924a5a7dc5b4,
STATUS_WITH_PORT=23:NORMAL,-3074457345618258603,
NATIVE_ADDRESS_AND_PORT=3:127.0.0.1:9042, RELEASE_VERSION=5:4.0.21-SNAPSHOT,
HOST_ID=2:2b443b7b-1998-43c3-8bc5-f65cea3e0eb2, SSTABLE_VERSIONS=6:big-nb,
DC=12:datacenter0}, /127.0.0.3={generation=1784034306, heartbeat=176,
LOAD=164:125932.0, NET_VERSION=1:12, TOKENS=22:<hidden>,
INTERNAL_ADDRESS_AND_PORT=8:127.0.0.3:7012, RACK=14:rack0,
SCHEMA=43:e8bb1f8c-f621-3f8b-aa57-924a5a7dc5b4,
STATUS_WITH_PORT=23:NORMAL,9223372036854775805,
NATIVE_ADDRESS_AND_PORT=3:127.0.0.3:9042, RELEASE_VERSION=5:4.0.21-SNAPSHOT,
HOST_ID=2:7512bf52-420b-46a9-a1c0-887144cf9fde, SSTABLE_VERSIONS=6:big-nb,
DC=12:datacenter0}}
at
org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossip(ClusterUtils.java:533)
at
org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossipStatus(ClusterUtils.java:547)
at
org.apache.cassandra.distributed.test.hostreplacement.BaseAssassinatedCase.test(BaseAssassinatedCase.java:82)
{code}
Interestingly, it does not fail on 5.0 (2).
Good news is that your newly introduced test passed the multiplexer (repeated
execution) which means your test is stable enough.
(1)
https://app.circleci.com/pipelines/github/instaclustr/cassandra/6564/workflows/6594a649-7388-43fd-9f66-d8da423d94d0/jobs/566218/tests#failed-test-0
(2) https://pre-ci.cassandra.apache.org/job/cassandra-5.0/164/
was (Author: smiklosovic):
[~curlylrt] thanks, from 4.0 CI (1) I see that
{{org.apache.cassandra.distributed.test.hostreplacement.AssassinateGracefullNodeTest}}
fails.
{code}
junit.framework.AssertionFailedError: Node node2 did not match state LEFT
{/127.0.0.2={generation=1784034306, RPC_READY=61:false, heartbeat=2147483647,
LOAD=28:26449.0, NET_VERSION=1:12, TOKENS=22:<hidden>,
INTERNAL_ADDRESS_AND_PORT=8:127.0.0.2:7012, RACK=14:rack0,
STATUS=60:shutdown,true, SCHEMA=43:e8bb1f8c-f621-3f8b-aa57-924a5a7dc5b4,
STATUS_WITH_PORT=59:shutdown,true, NATIVE_ADDRESS_AND_PORT=3:127.0.0.2:9042,
RELEASE_VERSION=5:4.0.21-SNAPSHOT,
HOST_ID=2:3ceff7f2-32fb-4167-9820-5d4b63ee72c2, SSTABLE_VERSIONS=6:big-nb,
DC=12:datacenter0}, /127.0.0.1={generation=1784034293,
INTERNAL_IP=10:127.0.0.1, heartbeat=188, LOAD=164:96368.0, NET_VERSION=1:12,
TOKENS=22:<hidden>, RPC_ADDRESS=4:127.0.0.1,
INTERNAL_ADDRESS_AND_PORT=8:127.0.0.1:7012, RACK=14:rack0,
STATUS=24:NORMAL,-3074457345618258603,
SCHEMA=58:e8bb1f8c-f621-3f8b-aa57-924a5a7dc5b4,
STATUS_WITH_PORT=23:NORMAL,-3074457345618258603,
NATIVE_ADDRESS_AND_PORT=3:127.0.0.1:9042, RELEASE_VERSION=5:4.0.21-SNAPSHOT,
HOST_ID=2:2b443b7b-1998-43c3-8bc5-f65cea3e0eb2, SSTABLE_VERSIONS=6:big-nb,
DC=12:datacenter0}, /127.0.0.3={generation=1784034306, heartbeat=176,
LOAD=164:125932.0, NET_VERSION=1:12, TOKENS=22:<hidden>,
INTERNAL_ADDRESS_AND_PORT=8:127.0.0.3:7012, RACK=14:rack0,
SCHEMA=43:e8bb1f8c-f621-3f8b-aa57-924a5a7dc5b4,
STATUS_WITH_PORT=23:NORMAL,9223372036854775805,
NATIVE_ADDRESS_AND_PORT=3:127.0.0.3:9042, RELEASE_VERSION=5:4.0.21-SNAPSHOT,
HOST_ID=2:7512bf52-420b-46a9-a1c0-887144cf9fde, SSTABLE_VERSIONS=6:big-nb,
DC=12:datacenter0}}
at
org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossip(ClusterUtils.java:533)
at
org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossipStatus(ClusterUtils.java:547)
at
org.apache.cassandra.distributed.test.hostreplacement.BaseAssassinatedCase.test(BaseAssassinatedCase.java:82)
{code}
Interestingly, it does not fail on 5.0 (2).
(1)
https://app.circleci.com/pipelines/github/instaclustr/cassandra/6564/workflows/6594a649-7388-43fd-9f66-d8da423d94d0/jobs/566218/tests#failed-test-0
(2) https://pre-ci.cassandra.apache.org/job/cassandra-5.0/164/
> nodetool assassinate blocks the GOSSIP stage, causing the executing node to
> be marked down and the liveness check to be ineffective
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21249
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21249
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip
> Reporter: Runtian Liu
> Assignee: Runtian Liu
> Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
>
> h2. Summary
> When {{nodetool assassinate}} is executed against a remote endpoint, two
> problems occur:
> The node running the command is marked as DOWN by other nodes in the cluster.
> Peers lose the ability to gossip with the executing node for ~34 seconds,
> causing the phi accrual failure detector to convict it.
> The liveness check is ineffective. The command is supposed to refuse to
> assassinate a live node by checking whether the target's heartbeat changes
> during a 30-second observation window. In practice, the check always passes
> trivially, meaning a fully healthy node can be assassinated without any
> warning.
> h2. Root Cause
> The {{assassinateEndpoint}} method in {{Gossiper.java}} runs its entire body
> — including a 30-second sleep ({{{}RING_DELAY{}}}) and a subsequent 4-second
> sleep — inside {{{}runInGossipStageBlocking(){}}}. This submits the work to
> {{{}Stage.GOSSIP{}}}, which is a single-threaded executor.
> While the ~34 seconds of sleep execute on the GOSSIP stage thread:
> All inbound gossip message processing is blocked. {{{}GossipDigestSyn{}}},
> {{{}GossipDigestAck{}}}, and {{GossipDigestAck2}} handlers are all dispatched
> to {{{}Stage.GOSSIP{}}}. They queue up and cannot execute.
> The failure detector on peer nodes receives no heartbeat updates from the
> executing node, because heartbeat reporting ({{{}notifyFailureDetector{}}})
> only happens during ACK/ACK2 processing — which is blocked. After ~10–20
> seconds of silence, peers convict the executing node as dead.
> The liveness check cannot detect a live target. The check compares the
> target's heartbeat before and after the 30-second sleep. However, the only
> code path that updates a remote endpoint's heartbeat on the local node is
> {{applyNewStates}} → {{{}setHeartBeatState{}}}, which runs on
> {{Stage.GOSSIP}} during gossip message processing — the very processing that
> is blocked. The heartbeat cannot change during the sleep, so the comparison
> always passes.
> h2. Regression Origin
> This was introduced by CASSANDRA-15059 , "Fix assorted gossip races and add
> related runtime checks", 2019-03-21).
> The commit's goal was to ensure that gossip state mutations only happen on
> the GOSSIP stage thread, adding {{runInGossipStageBlocking()}} wrappers
> throughout {{{}Gossiper{}}}. This was correct for methods like
> {{{}convict{}}}, {{{}markAsShutdown{}}}, {{{}removeEndpoint{}}}, and
> {{{}evictFromMembership{}}}. However, wrapping the entire
> {{assassinateEndpoint}} moved the long-duration sleeps onto the GOSSIP stage
> as an unintended side effect.
> Before CASSANDRA-15059: {{assassinateEndpoint}} ran entirely on the JMX
> thread. The GOSSIP stage remained free to process messages during the
> 30-second sleep, heartbeat updates from the target were visible, and the
> liveness check correctly detected live nodes.
> After CASSANDRA-15059: The entire method body runs on the GOSSIP stage. The
> liveness check is broken and the executing node gets convicted.
> h2. Affected Versions
> All versions carrying CASSANDRA-15059 that use gossip-based assassination:
> 3.0.28+, 3.11.x, 4.0.x, 4.1.x, 5.0.x. Trunk (5.1+) is not affected because
> gossip-based assassination was replaced by the TCM system.
> h2. Proposed Fix
> Split {{assassinateEndpoint}} into phases so that only the state mutations
> run on the GOSSIP stage, while reads and sleeps run on the caller (JMX)
> thread:
> Phase 1 (JMX thread): Snapshot the target's heartbeat generation and version.
> This is safe off the GOSSIP stage because {{endpointStateMap}} is a
> {{ConcurrentHashMap}} and {{HeartBeatState}} fields are {{{}volatile{}}}.
> Phase 2 (JMX thread): Sleep for {{RING_DELAY}} — the GOSSIP stage remains
> free to process messages, so if the target is alive, its heartbeat updates
> will be received.
> Phase 3 (GOSSIP stage): Re-read the target's heartbeat. If generation or
> version changed, throw {{{}RuntimeException("Endpoint still alive"){}}}.
> Otherwise, perform the assassination (set status to LEFT, call
> {{{}handleMajorStateChange{}}}).
> Phase 4 (JMX thread): Sleep for {{intervalInMillis * 4}} to allow gossip
> propagation.
> This restores the pre-CASSANDRA-15059 behavior: the GOSSIP stage is only held
> for the brief duration of state mutations in Phase 3, the liveness check
> works because heartbeat updates are processed during the sleep, and the
> executing node is not convicted by peers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
