[
https://issues.apache.org/jira/browse/CASSANDRA-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated CASSANDRA-21097:
------------------------------
Description:
The `ClusterUtils.parseGossipInfo` method parses the gossip info output from
`nodetool gossipinfo` by checking the line starts with "/". It expects endpoint
addresses to start with "/".
This assumption might be broken in the future with the following workflow, let
me use an existing example to show you.
In the existing test {*}JMXGetterCheckTest.testAllValidGetters(){*}, it call
ALL JMX attributes including:
{code:java}
// FailureDetectorMBean methods that trigger hostname resolution:
getAllEndpointStatesWithResolveIp() // resolveIp=true
getAllEndpointStatesWithPortAndResolveIp() // resolveIp=true {code}
These methods call `entry.getKey().getHostName()` on the gossip endpoint
addresses (`FailureDetector.java:149`), which caches the hostname in the
underlying `InetAddress` objects.
Then:
{code:java}
1. Test starts, cluster initializes
└── Gossip endpoints created with InetAddress (hostname NOT resolved)
└── InetAddress.toString() → "/127.0.0.1"
2. JMXGetterCheckTest calls all JMX getters
└── Calls getAllEndpointStatesWithResolveIp()
└── This calls getHostName() on gossip endpoint InetAddress objects
└── Hostname "localhost" gets CACHED in the InetAddress objects
3. Node restart occurs
└── Restart adapter calls awaitGossipSchemaMatch()
└── This runs "nodetool gossipinfo" (resolveIp=false)
└── But InetAddress.toString() now returns "localhost/127.0.0.1" (cached!)
4. parseGossipInfo() fails
└── Checks: line.startsWith("/") → FALSE for "localhost/127.0.0.1"
└── currentInstance remains null
└── Objects.requireNonNull(currentInstance) → NPE! {code}
Code that checks line start with "/":
{code:java}
private static Map<String, Map<String, String>> parseGossipInfo(String str)
{
Map<String, Map<String, String>> map = new HashMap<>();
String[] lines = str.split("\n");
String currentInstance = null;
for (String line : lines)
{
if (line.startsWith("/")) // BUG: This condition is incorrect
{
// start of new instance
currentInstance = line;
continue;
}
Objects.requireNonNull(currentInstance); // NPE thrown here
String[] kv = line.trim().split(":", 2);
assert kv.length == 2 : "When splitting line '" + line + "' expected 2
parts but not true";
Map<String, String> state = map.computeIfAbsent(currentInstance, ignore
-> new HashMap<>());
state.put(kv[0], kv[1]);
}
return map;
} {code}
Java's `InetAddress` caches hostname at the *object level.* The gossip
`endpointStateMap` contains shared `InetAddressAndPort` objects. Once ANY code
path calls `getHostName()` on these objects, all subsequent `toString()` calls
return the hostname.
h2. Stacktrace
{code:java}
java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:209)
at
org.apache.cassandra.distributed.shared.ClusterUtils.parseGossipInfo(ClusterUtils.java:691)
at
org.apache.cassandra.distributed.shared.ClusterUtils.gossipInfo(ClusterUtils.java:655)
at
org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossip(ClusterUtils.java:554)
at
org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossipSchemaMatch(ClusterUtils.java:600)
{code}
was:
The `ClusterUtils.parseGossipInfo` method parses the gossip info output from
`nodetool gossipinfo` by checking the line starts with "/". It expects endpoint
addresses to start with "/".
But when there is a node restart, the actual format from Java's
`InetAddress.toString()` is changed to "hostname/ipaddress" (e.g.,
`localhost/127.0.0.1`).
{code:java}
private static Map<String, Map<String, String>> parseGossipInfo(String str)
{
Map<String, Map<String, String>> map = new HashMap<>();
String[] lines = str.split("\n");
String currentInstance = null;
for (String line : lines)
{
if (line.startsWith("/")) // BUG: This condition is incorrect
{
// start of new instance
currentInstance = line;
continue;
}
Objects.requireNonNull(currentInstance); // NPE thrown here
String[] kv = line.trim().split(":", 2);
assert kv.length == 2 : "When splitting line '" + line + "' expected 2
parts but not true";
Map<String, String> state = map.computeIfAbsent(currentInstance, ignore
-> new HashMap<>());
state.put(kv[0], kv[1]);
}
return map;
} {code}
The method checks `if (line.startsWith("/"))` to identify endpoint address
lines. However, the actual gossip info output format after a node restart looks
like:
{code:java}
localhost/127.0.0.1 <-- Does NOT start with "/"
127.0.0.2/127.0.0.2 <-- Does NOT start with "/"{code}
The Java documentation for `InetAddress.toString()` states:
{code:java}
> "The string is of the form: hostname / literal IP address"{code}
h2. Actual Gossip Info Output after node restart
{code:java}
localhost/127.0.0.1
generation:1767679555
heartbeat:74
STATUS:23:NORMAL,-1
LOAD:27:144545.0
SCHEMA:17:d03783d7-b468-3c1a-82f1-8e30b2edde8b
DC:13:datacenter0
RACK:15:rack0
...
127.0.0.2/127.0.0.2
generation:1767679591
heartbeat:33
... {code}
The first line `localhost/127.0.0.1` does NOT start with "/" so it's not
recognized as an endpoint. When the method tries to process the next line `
generation:1767679555`, `currentInstance` is still null, causing the NPE.
h2. Stacktrace
{code:java}
java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:209)
at
org.apache.cassandra.distributed.shared.ClusterUtils.parseGossipInfo(ClusterUtils.java:691)
at
org.apache.cassandra.distributed.shared.ClusterUtils.gossipInfo(ClusterUtils.java:655)
at
org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossip(ClusterUtils.java:554)
at
org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossipSchemaMatch(ClusterUtils.java:600)
{code}
> ClusterUtils.parseGossipInfo Fails to Parse Gossip Output Format after a node
> restart
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21097
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21097
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Test/dtest/java
> Reporter: ConfX
> Priority: Normal
>
> The `ClusterUtils.parseGossipInfo` method parses the gossip info output from
> `nodetool gossipinfo` by checking the line starts with "/". It expects
> endpoint addresses to start with "/".
> This assumption might be broken in the future with the following workflow,
> let me use an existing example to show you.
> In the existing test {*}JMXGetterCheckTest.testAllValidGetters(){*}, it call
> ALL JMX attributes including:
>
> {code:java}
> // FailureDetectorMBean methods that trigger hostname resolution:
> getAllEndpointStatesWithResolveIp() // resolveIp=true
> getAllEndpointStatesWithPortAndResolveIp() // resolveIp=true {code}
> These methods call `entry.getKey().getHostName()` on the gossip endpoint
> addresses (`FailureDetector.java:149`), which caches the hostname in the
> underlying `InetAddress` objects.
>
> Then:
>
> {code:java}
> 1. Test starts, cluster initializes
> └── Gossip endpoints created with InetAddress (hostname NOT resolved)
> └── InetAddress.toString() → "/127.0.0.1"
> 2. JMXGetterCheckTest calls all JMX getters
> └── Calls getAllEndpointStatesWithResolveIp()
> └── This calls getHostName() on gossip endpoint InetAddress objects
> └── Hostname "localhost" gets CACHED in the InetAddress objects
> 3. Node restart occurs
> └── Restart adapter calls awaitGossipSchemaMatch()
> └── This runs "nodetool gossipinfo" (resolveIp=false)
> └── But InetAddress.toString() now returns "localhost/127.0.0.1" (cached!)
> 4. parseGossipInfo() fails
> └── Checks: line.startsWith("/") → FALSE for "localhost/127.0.0.1"
> └── currentInstance remains null
> └── Objects.requireNonNull(currentInstance) → NPE! {code}
>
>
> Code that checks line start with "/":
> {code:java}
> private static Map<String, Map<String, String>> parseGossipInfo(String str)
> {
> Map<String, Map<String, String>> map = new HashMap<>();
> String[] lines = str.split("\n");
> String currentInstance = null;
> for (String line : lines)
> {
> if (line.startsWith("/")) // BUG: This condition is incorrect
> {
> // start of new instance
> currentInstance = line;
> continue;
> }
> Objects.requireNonNull(currentInstance); // NPE thrown here
> String[] kv = line.trim().split(":", 2);
> assert kv.length == 2 : "When splitting line '" + line + "' expected
> 2 parts but not true";
> Map<String, String> state = map.computeIfAbsent(currentInstance,
> ignore -> new HashMap<>());
> state.put(kv[0], kv[1]);
> }
> return map;
> } {code}
> Java's `InetAddress` caches hostname at the *object level.* The gossip
> `endpointStateMap` contains shared `InetAddressAndPort` objects. Once ANY
> code path calls `getHostName()` on these objects, all subsequent `toString()`
> calls return the hostname.
> h2. Stacktrace
> {code:java}
> java.lang.NullPointerException
> at java.base/java.util.Objects.requireNonNull(Objects.java:209)
> at
> org.apache.cassandra.distributed.shared.ClusterUtils.parseGossipInfo(ClusterUtils.java:691)
> at
> org.apache.cassandra.distributed.shared.ClusterUtils.gossipInfo(ClusterUtils.java:655)
> at
> org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossip(ClusterUtils.java:554)
> at
> org.apache.cassandra.distributed.shared.ClusterUtils.awaitGossipSchemaMatch(ClusterUtils.java:600)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
