Thanks Robert!
Yes I tried what you said: clean the data and re-bootstrap. But still it
failed, once at the point of 600GB transferred and once at 1.1TB :(
But I could see following exceptions from time to time:
=====================
java.io.IOException: net.jpountz.lz4.LZ4Exception: Error decoding offset 15
of input buffer
at
org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:89)
at
org.apache.cassandra.streaming.compress.CompressedInputStream.decompress(CompressedInputStream.java:108)
at
org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:86)
at java.io.InputStream.read(InputStream.java:170)
at java.io.InputStream.skip(InputStream.java:222)
at
org.apache.cassandra.streaming.StreamReader.drain(StreamReader.java:117)
at
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:89)
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:47)
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:37)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:283)
at java.lang.Thread.run(Thread.java:744)
=======================
And
=======================
CassandraDaemon.java (line 479) Exception encountered during startup
java.lang.RuntimeException: Error during boostrap: Stream failed
at
org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:975)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:736)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:583)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:482)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:345)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
at
com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:329)
at
org.apache.cassandra.streaming.StreamSession.convict(StreamSession.java:592)
at
org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:236)
at
org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:623)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:64)
at
org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:170)
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:75)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
INFO [StorageServiceShutdownHook] 2015-03-26 10:29:48,471 Gossiper.java
(line 1251) Announcing shutdown
==========================
Is there anything else I could try?
Thanks!
Shiwen
On Thu, Mar 26, 2015 at 4:18 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Mar 26, 2015 at 11:31 AM, Shiwen Cheng <cheng.shiwen...@gmail.com>
> wrote:
>
>> I encountered an issue by removing and adding back a node.
>>
>
> You are encountering a failed/hung bootstrap, which probably has nothing
> to do with the node having been previously removenoded.
>
> Stop the node, wipe all the data on the node, including it's system
> directory and re-bootstrap.
>
> =Rob
>
>
