https://issues.apache.org/jira/browse/CASSANDRA-11840
increase streaming_socket_timeout to 86400000 or upgrade to cassandra-2.1.15. 2016-08-05 12:28 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: > > Hello Paulo, > > Thx for your fast replay. > > You are right about that node, I did not see it that fast. In this node we > have errors of .SocketTimeoutException: null > > > > ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456 > StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8] > Streaming error occurred > java.net.SocketTimeoutException: null > at > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) > ~[na:1.8.0_60] > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > ~[na:1.8.0_60] > at > java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) > ~[na:1.8.0_60] > at org.apache.cassandra.streaming.messages. > StreamMessage.deserialize(StreamMessage.java:51) > ~[apache-cassandra-2.1.14.jar:2.1.14] > at org.apache.cassandra.streaming.ConnectionHandler$ > IncomingMessageHandler.run(ConnectionHandler.java:257) > ~[apache-cassandra-2.1.14.jar:2.1.14] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] > > We have this error with at least 5 more nodes in the same log. But the > error doesn't say that much > > > Saludos > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > > On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pauloricard...@gmail.com> > wrote: > >> you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS >> >> 2016-08-05 12:08 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: >> >>> Hi Paulo >>> >>> I found the lines, we got an exception "Outgoing stream handler has been >>> closed" these are >>> >>> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 >>> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>> Remote peer 192.168.0.36 failed stream session. >>> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 >>> StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>> Session with /192.168.0.36 is complete >>> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773 >>> StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>> Stream failed >>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 >>> StreamReceiveTask.java:183 - Error applying streamed data: >>> java.lang.RuntimeException: Outgoing stream handler has been closed >>> at >>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>> at >>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) >>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>> at >>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) >>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio >>> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar: >>> 2.1.14] >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> [na:1.8.0_60] >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> [na:1.8.0_60] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>> >>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 >>> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>> Streaming error occurred >>> java.lang.RuntimeException: Outgoing stream handler has been closed >>> at >>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>> at >>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) >>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>> at >>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) >>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio >>> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar: >>> 2.1.14] >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> [na:1.8.0_60] >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> [na:1.8.0_60] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>> >>> I al looking through the changes files to see if it is a bug fixed in >>> the 2.1.15 >>> >>> >>> >>> Saludos >>> >>> Jean Carlo >>> >>> "The best way to predict the future is to invent it" Alan Kay >>> >>> On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pauloricard...@gmail.com> >>> wrote: >>> >>>> It seems you have a streaming error, look for ERROR statement in the >>>> streaming classes before that which may give you a more specific root >>>> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a >>>> couple of streaming fixes on this version that might help. >>>> >>>> 2016-08-05 11:15 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: >>>> >>>>> >>>>> Hi guys, Doing a repair I got this error for 2 tokenranges. >>>>> >>>>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288 >>>>> StorageService.java:3068 - Repair session >>>>> 41e4bab0-5a63-11e6-9993-e11d93fd5b40 >>>>> for range (487410372471205090,492009442088088379] failed with error >>>>> org.apache.cassandra.exceptions.RepairExcep >>>>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >>>>> (487410372471205090,492009442088088379]] Sync failed between / >>>>> 192.168.0.144 and /192.168.0.37 >>>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException: >>>>> org.apache.cassandra.exceptions.RepairException: [repair >>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >>>>> (487410372471205090,492009442088088379]] Sync failed between / >>>>> 192.168.0.144 and /192.168.0.37 >>>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >>>>> [na:1.8.0_60] >>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >>>>> [na:1.8.0_60] >>>>> at org.apache.cassandra.service.S >>>>> torageService$4.runMayThrow(StorageService.java:3059) >>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>> at org.apache.cassandra.utils.Wra >>>>> ppedRunnable.run(WrappedRunnable.java:28) >>>>> [apache-cassandra-2.1.14.jar:2.1.14] >>>>> at java.util.concurrent.Executors >>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> [na:1.8.0_60] >>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>>>> Caused by: java.lang.RuntimeException: >>>>> org.apache.cassandra.exceptions.RepairException: >>>>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >>>>> (487410372471205090,492009442088088379]] Sync failed between / >>>>> 10.234.72.144 and >>>>> /10.234.86.37 >>>>> at com.google.common.base.Throwab >>>>> les.propagate(Throwables.java:160) ~[guava-16.0.jar:na] >>>>> at org.apache.cassandra.utils.Wra >>>>> ppedRunnable.run(WrappedRunnable.java:32) >>>>> [apache-cassandra-2.1.14.jar:2.1.14] >>>>> at java.util.concurrent.Executors >>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> [na:1.8.0_60] >>>>> at java.util.concurrent.ThreadPoo >>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60] >>>>> at java.util.concurrent.ThreadPoo >>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60] >>>>> ... 1 common frames omitted >>>>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair >>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >>>>> (487410372471205090,492009442088088379]] Sync failed between / >>>>> 192.168.0.144 and /192.168.0.37 >>>>> at org.apache.cassandra.repair.Re >>>>> pairSession.syncComplete(RepairSession.java:223) >>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>> at org.apache.cassandra.service.A >>>>> ctiveRepairService.handleMessage(ActiveRepairService.java:422) >>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>> at org.apache.cassandra.repair.Re >>>>> pairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134) >>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>> at org.apache.cassandra.net.Messa >>>>> geDeliveryTask.run(MessageDeliveryTask.java:64) >>>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>>> ... 3 common frames omitted >>>>> >>>>> Is it a network problem or we are having a bug ? There is nothing else >>>>> in the log of cassandra more than >>>>> Sync failed between. >>>>> >>>>> We are having cassandra 2.1.14 >>>>> >>>>> Best regards >>>>> >>>>> Jean Carlo >>>>> >>>>> "The best way to predict the future is to invent it" Alan Kay >>>>> >>>> >>>> >>> >> >
