Hi All,
I see a different error as below this time:
WARN [STREAM-IN-/***] 2019-12-21 19:42:11,662 StreamSession.java:641 - [Stream
#ec8d1000-2451-11ea-b38a-0f6cfab62953] Retrying for following error
java.lang.ArrayIndexOutOfBoundsException: null
ERROR [STREAM-IN-/***] 2019-12-21 19:42:12,034 StreamSession.java:520 - [Stream
#ec8d1000-2451-11ea-b38a-0f6cfab62953] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at
org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [STREAM-IN-/10.146.128.41] 2019-12-21 19:42:12,037
StreamResultFuture.java:185 - [Stream #ec8d1000-2451-11ea-b38a-0f6cfab62953]
Session with /*** is complete
WARN [STREAM-IN-/10.146.128.41] 2019-12-21 19:42:12,038
StreamResultFuture.java:212 - [Stream #ec8d1000-2451-11ea-b38a-0f6cfab62953]
Stream failed
WARN [Thread-862] 2019-12-21 19:42:12,039 CompressedInputStream.java:182 -
Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
at
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
~[na:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
~[na:1.8.0_74]
at
org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Does anyone know a resolution for this ?
From: Jon Haddad <j...@jonhaddad.com>
Sent: Friday, December 20, 2019 2:29 PM
To: user@cassandra.apache.org
Subject: Re: Streaming Failed during bootstrap of a Replacement node
This email is from an external source - exercise caution regarding links and
attachments.
Without getting too in the weeds here - ideally yes, you'd be able to replace
the node. However, the error you're getting looks (at a glance) to be one of
the many range tombstone bugs that were fixed in later versions.
On Fri, Dec 20, 2019 at 11:23 AM Nethi, Manoj
<manoj.ne...@fmr.com.invalid<mailto:manoj.ne...@fmr.com.invalid>> wrote:
Hi Jon,
Yes we will upgrade it soon. But before we can upgrade shouldn’t we get this
lost node in the cluster to be replaced ?
From: Jon Haddad <j...@jonhaddad.com<mailto:j...@jonhaddad.com>>
Sent: Friday, December 20, 2019 2:13 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Streaming Failed during bootstrap of a Replacement node
This email is from an external source - exercise caution regarding links and
attachments.
You should upgrade to Cassandra 3.11.5 before doing anything else. You're
running a pretty old and buggy version. There's been hundreds (maybe
thousands) of bugs fixed between 3.3 and 3.11.5.
On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj
<manoj.ne...@fmr.com.invalid<mailto:manoj.ne...@fmr.com.invalid>> wrote:
Hi,
We are seeing the following error while bootstrapping a node which is
replacement of a failed node in a multi DC cluster.
WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:641
- [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Retrying for following error
java.lang.ArrayIndexOutOfBoundsException: -127
at
org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:520
- [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at
org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
StreamResultFuture.java:185 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
Session with /**.***.**.** is complete
WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
StreamResultFuture.java:212 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
Stream failed
WARN [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182 -
Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
at
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
~[na:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
~[na:1.8.0_74]
at
org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Did any one faced a similar issue ? What should be done to resolve this ?
Thanks
Manoj
