Hi all,
Recently I face some issues when a node is joining an already existing cluster,
the bootstrap ends up incomplete and in failed state due to some streams have
failed.
Using Cassandra 5.0.3 running on OpenJDK 17.0.15:
Logs show:
WARN [NonPeriodicTasks:1] 2025-05-08 13:23:26,145 StreamResultFuture.java:250
- [Stream #e8dc7090-2bf8-11f0-8e15-57fc8d65eca3] Stream failed:
Session peer /172.20.145.155:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.29:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.207:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.209:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
ERROR [main] 2025-05-08 13:23:26,149 StorageService.java:2216 - Error while
waiting on bootstrap to complete. Bootstrap will have to
be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed:
Session peer /172.20.145.155:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.29:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.207:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.209:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
at
org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
at
org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2209)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1292)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1252)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:1031)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:947)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:383)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed:
Session peer /172.20.145.155:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.29:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.207:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.209:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot
be cast to class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In
compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:251)
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
at
org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:552)
at
org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:124)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
INFO [main] 2025-05-08 13:23:26,149 StorageService.java:2042 - JOINING_FAILED
WARN [main] 2025-05-08 13:23:26,180 StorageService.java:1328 - Some data
streaming failed. Use nodetool to check bootstrap state an
d resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2025-05-08 13:23:26,189 Gossiper.java:2449 - Waiting for gossip to
settle...
INFO [main] 2025-05-08 13:23:34,190 Gossiper.java:2480 - No gossip backlog;
proceeding
Any clue what’s going wrong here? The part with
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader ‘app’ looks very strange.
