[
https://issues.apache.org/jira/browse/CASSANDRA-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522129#comment-14522129
]
Blake Eggleston commented on CASSANDRA-6702:
--------------------------------------------
Same problem upgrading from 1.1.x to 2.0.x.
The crux of the issue is here:
https://github.com/apache/cassandra/blob/cassandra-1.2/src/java/org/apache/cassandra/net/IncomingTcpConnection.java#L129
When an upgraded node connects to a non-upgraded node, that call to
Gossiper.instance.addSavedNode marks the connecting node as down and forces it
to reconnect. That's usually not a problem, except when using the
Ec2MultiRegionSnitch, since the ReconnectableSnitchHelper reconnects
onJoin/onChange/onAlive, which starts a cycle of reconnection. So this was
worked around in CASSANDRA-5669 by just skipping the reconnect step if the
cluster was upgrading here:
https://github.com/apache/cassandra/blob/cassandra-1.2/src/java/org/apache/cassandra/locator/ReconnectableSnitchHelper.java#L65
However for upgrades from versions >= 1.2 to versions >= 1.2, I think we should
be able to skip that gossip call, since MS protocol version is no longer
tracked by gossip as of 1.2. If the call was just making sure we reconnected,
we should be able to replace it with a call to
MessagingService.instance.getConnectionPool for the same effect.
With that out of the way, we should be able to remove the version check from
ReconnectableSnitchHelper. That should probably go behind a -D flag until 3.0
though, since all hell will break loose if someone using the
Ec2MultiRegionSnitch upgrades from a release without that fix
> Upgrading node uses the wrong port in gossiping
> -----------------------------------------------
>
> Key: CASSANDRA-6702
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6702
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 1.1.7, AWS, Ec2MultiRegionSnitch
> Reporter: Minh Do
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 2.0.x
>
>
> When upgrading a node in 1.1.7 (or 1.1.11) cluster to 1.2.15 and inspecting
> the gossip information on port/Ip, I could see that the upgrading node (1.2
> version) communicates to one other node in the same region using Public IP
> and non-encrypted port.
> For the rest, the upgrading node uses the correct ports and IPs to
> communicate in this manner:
> Same region: private IP and non-encrypted port
> and
> Different region: public IP and encrypted port
> Because there is one node like this (or 2 out of 12 nodes cluster in which
> nodes are split equally on 2 AWS regions), we have to modify Security Group
> to allow the new traffics.
> Without modifying the SG, the 95th and 99th latencies for both reads and
> writes in the cluster are very bad (due to RPC timeout). Inspecting closer,
> that upgraded node (1.2 node) is contributing to all of the high latencies
> whenever it acts as a coordinator node.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
