Without getting too in the weeds here - ideally yes, you'd be able to replace the node. However, the error you're getting looks (at a glance) to be one of the many range tombstone bugs that were fixed in later versions.
On Fri, Dec 20, 2019 at 11:23 AM Nethi, Manoj <manoj.ne...@fmr.com.invalid> wrote: > Hi Jon, > > Yes we will upgrade it soon. But before we can upgrade shouldn’t we get > this lost node in the cluster to be replaced ? > > > > > > > > *From:* Jon Haddad <j...@jonhaddad.com> > *Sent:* Friday, December 20, 2019 2:13 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Streaming Failed during bootstrap of a Replacement node > > > > *This email is from an external source - **exercise caution regarding > links and attachments. * > > > > You should upgrade to Cassandra 3.11.5 before doing anything else. You're > running a pretty old and buggy version. There's been hundreds (maybe > thousands) of bugs fixed between 3.3 and 3.11.5. > > > > On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj <manoj.ne...@fmr.com.invalid> > wrote: > > Hi, > > > > We are seeing the following error while bootstrapping a node which is > replacement of a failed node in a multi DC cluster. > > > > > > > > WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 > StreamSession.java:641 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] > Retrying for following error > > java.lang.ArrayIndexOutOfBoundsException: -127 > > at > org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50) > [apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39) > [apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59) > [apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) > [apache-cassandra-3.3.0.jar:3.3.0] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] > > ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 > StreamSession.java:520 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] > Streaming error occurred > > java.lang.IllegalArgumentException: Unknown type 0 > > at > org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] > > INFO [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 > StreamResultFuture.java:185 - [Stream > #f231db30-22da-11ea-b38a-0f6cfab62953] Session with /**.***.**.** is > complete > > WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 > StreamResultFuture.java:212 - [Stream > #f231db30-22da-11ea-b38a-0f6cfab62953] Stream failed > > WARN [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182 > - Error while reading compressed input stream. > > java.nio.channels.ClosedChannelException: null > > at > sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) > ~[na:1.8.0_74] > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) > ~[na:1.8.0_74] > > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) > ~[na:1.8.0_74] > > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) > ~[na:1.8.0_74] > > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > ~[na:1.8.0_74] > > at > org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172) > ~[apache-cassandra-3.3.0.jar:3.3.0] > > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > [apache-cassandra-3.3.0.jar:3.3.0] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] > > > > > > > > Did any one faced a similar issue ? What should be done to resolve this ? > > > > > > > > Thanks > > Manoj > >
