[
https://issues.apache.org/jira/browse/CASSANDRA-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18024300#comment-18024300
]
Brandon Williams edited comment on CASSANDRA-20910 at 10/2/25 3:10 PM:
-----------------------------------------------------------------------
bq. not sure if this helps but we noticed the following Prometheus ERROR:
The thing I notice here is the thread pool name `prometheus-netty-pool-0` since
C* doesn't integrate prometheus, where is this coming from?
Maybe specifying the seed port helps keep things on the right track, but we
still don't understand why the cluster name mismatch didn't protect there, so
it's hard to say whether this will prevent anything. I think explicit is
better than implicit though, so it's a good change to have regardless.
was (Author: brandon.williams):
> not sure if this helps but we noticed the following Prometheus ERROR:
The thing I notice here is the thread pool name `prometheus-netty-pool-0` since
C* doesn't integrate prometheus, where is this coming from?
Maybe specifying the seed port helps keep things on the right track, but we
still don't understand why the cluster name mismatch didn't protect there, so
it's hard to say whether this will prevent anything. I think explicit is
better than implicit though, so it's a good change to have regardless.
> Instances from a 2nd ring join another ring when running on the same nodes
> ---------------------------------------------------------------------------
>
> Key: CASSANDRA-20910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20910
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Chris Miller
> Priority: Urgent
>
> Hi,
> We experienced an issue today whereby instances from a 2nd ring join another
> ring when running on the same nodes following a rolling restart which took
> place following an OS patch and node reboot (both on Cassandra 4.1.2).
> The cluster names and storage ports are different and this type of activity
> normally runs without issue.
> Any ideas as to what could have happened? Could this be a bug?
> The seeds use the same IP addresses but no storage port is configured in the
> seeds parameter, should we add the storage port to prevent this from
> happening again? Any thoughts?
> Messages like the following could be seen on ring 1.
> INFO [GossipStage:1] 2025-09-18 04:11:49,040 Gossiper.java:1434 - Node
> /XX.XX.XX.190:7002 is now part of the cluster
> INFO [GossipStage:1] 2025-09-18 04:11:49,043 TokenMetadata.java:539 -
> Updating topology for /XX.XX.XX.190:7002
> INFO [Messaging-EventLoop-3-8] 2025-09-18 04:11:49,044
> OutboundConnection.java:1153 -
> /XX.XX.XX.61:7000(/XX.XX.XX.61:41920)->/XX.XX.XX.190:7002-URGENT_MESSAGES-7af53583
> successfully connected, version = 12, framing = CRC, encryption = unencrypted
> INFO [GossipStage:1] 2025-09-18 04:11:49,044 TokenMetadata.java:539 -
> Updating topology for /XX.XX.XX.190:7002
> INFO [GossipStage:1] 2025-09-18 04:11:49,044 Gossiper.java:1434 - Node
> /XX.XX.XX.214:7002 is now part of the cluster
> INFO [Messaging-EventLoop-3-3] 2025-09-18 04:11:49,046
> OutboundConnection.java:1153 -
> /XX.XX.XX.61:7000(/XX.XX.XX.61:62628)->/XX.XX.XX.214:7002-URGENT_MESSAGES-0515b24a
> successfully connected, version = 12, framing = CRC, encryption = unencrypted
> INFO [GossipStage:1] 2025-09-18 04:11:49,046 TokenMetadata.java:539 -
> Updating topology for /XX.XX.XX.214:7002
> INFO [GossipStage:1] 2025-09-18 04:11:49,046 TokenMetadata.java:539 -
> Updating topology for /XX.XX.XX.214:7002
> INFO [GossipStage:1] 2025-09-18 04:11:49,047 Gossiper.java:1434 - Node
> /XX.XX.XX.247:7002 is now part of the cluster
> INFO [Messaging-EventLoop-3-4] 2025-09-18 04:11:49,048
> InboundConnectionInitiator.java:529 -
> /XX.XX.XX.190:7002(/XX.XX.XX.190:60180)->/XX.XX.XX.61:7000-URGENT_MESSAGES-edfb2d8f
> messaging connection established, version = 12, framing = LZ4, encryption =
> unencrypted
> Messages like the following in ring 2:
> WARN [GossipStage:1] 2025-09-18 04:11:49,304
> GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
> /XX.XX.XX.247:7000 ring1!=ring2
> WARN [GossipStage:1] 2025-09-18 04:11:49,819
> GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
> /XX.XX.XX.108:7000 ring1!=ring2
> WARN [GossipStage:1] 2025-09-18 04:11:51,598
> GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
> /XX.XX.XX.190:7000 ring1!=ring2
> WARN [GossipStage:1] 2025-09-18 04:11:52,361
> GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
> /XX.XX.XX.111:7000 ring1!=ring2
> WARN [GossipStage:1] 2025-09-18 04:11:53,489
> GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
> /XX.XX.XX.84:7000 ring1!=ring2
> WARN [GossipStage:1] 2025-09-18 04:11:58,322
> GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
> /XX.XX.XX.247:7000 ring1!=ring2
> Instances from ring2 were listed in nodetool describecluster as unreachable
> under schema versions.
> They were also listed as DN under nodetool status.
> The nodetool removenode command was used to remove the instances successfully.
> Regards,
> Chris.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
