>From the various stacktraces in this thread, it's obvious you are mixing versions 2.1.13 and 2.1.14. Topology changes like this aren't supported with mixed Cassandra versions. Sometimes it will work, sometimes it won't (and it will definitely not work in this instance).
You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add the new nodes using 2.1.13, and upgrade after. On Fri, May 27, 2016 at 8:41 AM, George Sigletos <sigle...@textkernel.nl> wrote: >>>> ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027 >>>> StreamSession.java:505 - [Stream #74c57bc0-231a-11e6-a698-1b05ac77baf9] >>>> Streaming error occurred >>>> java.lang.RuntimeException: Outgoing stream handler has been closed >>>> at >>>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79] >>>> >>>> And this is from the source node: >>>> >>>> ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097 >>>> StreamSession.java:505 - [Stream #74c57bc0-231a-11e6-a698-1b05ac77baf9] >>>> Streaming error occurred >>>> java.io.IOException: Broken pipe >>>> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) >>>> ~[na:1.7.0_79] >>>> at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source) >>>> ~[na:1.7.0_79] >>>> at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source) >>>> ~[na:1.7.0_79] >>>> at >>>> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358) >>>> [apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330) >>>> [apache-cassandra-2.1.14.jar:2.1.14] >>>>>>>>>>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704 >>>>>>>>>>> StreamSession.java:620 - [Stream >>>>>>>>>>> #2c290460-20d4-11e6-930f-1b05ac77baf9] >>>>>>>>>>> Remote peer 192.168.1.140 failed stream session. >>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:57,705 >>>>>>>>>>> StreamSession.java:505 - [Stream >>>>>>>>>>> #2c290460-20d4-11e6-930f-1b05ac77baf9] >>>>>>>>>>> Streaming error occurred >>>>>>>>>>> java.io.IOException: Connection timed out >>>>>>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>>>>>>>>> ~[na:1.7.0_79] >>>>>>>>>>> at sun.nio.ch.SocketDispatcher.write(Unknown Source) >>>>>>>>>>> ~[na:1.7.0_79] >>>>>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown >>>>>>>>>>> Source) ~[na:1.7.0_79] >>>>>>>>>>> at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_79] >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source) >>>>>>>>>>> ~[na:1.7.0_79] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351) >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323) >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79] >>>>>>>>>>> INFO [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,625 >>>>>>>>>>> StreamResultFuture.java:180 - [Stream >>>>>>>>>>> #2c290460-20d4-11e6-930f-1b05ac77baf9] >>>>>>>>>>> Session with /192.168.1.140 is complete >>>>>>>>>>> WARN [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,627 >>>>>>>>>>> StreamResultFuture.java:207 - [Stream >>>>>>>>>>> #2c290460-20d4-11e6-930f-1b05ac77baf9] >>>>>>>>>>> Stream failed >>>>>>>>>>> ERROR [RMI TCP Connection(24)-127.0.0.1] 2016-05-24 22:44:58,628 >>>>>>>>>>> StorageService.java:1075 - Error while rebuilding node >>>>>>>>>>> org.apache.cassandra.streaming.StreamException: Stream failed >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) >>>>>>>>>>> ~[guava-16.0.jar:na] >>>>>>>>>>> at >>>>>>>>>>> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) >>>>>>>>>>> ~[guava-16.0.jar:na] >>>>>>>>>>> at >>>>>>>>>>> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) >>>>>>>>>>> ~[guava-16.0.jar:na] >>>>>>>>>>> at >>>>>>>>>>> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) >>>>>>>>>>> ~[guava-16.0.jar:na] >>>>>>>>>>> at >>>>>>>>>>> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) >>>>>>>>>>> ~[guava-16.0.jar:na] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:415) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:621) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:475) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:256) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_79] >>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:58,629 >>>>>>>>>>> StreamSession.java:505 - [Stream >>>>>>>>>>> #2c290460-20d4-11e6-930f-1b05ac77baf9] >>>>>>>>>>> Streaming error occurred >>>>>>>>>>> java.io.IOException: Broken pipe >>>>>>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >>>>>>>>>>> ~[na:1.7.0_79] >>>>>>>>>>> at sun.nio.ch.SocketDispatcher.write(Unknown Source) >>>>>>>>>>> ~[na:1.7.0_79] >>>>>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown >>>>>>>>>>> Source) ~[na:1.7.0_79] >>>>>>>>>>> at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_79] >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source) >>>>>>>>>>> ~[na:1.7.0_79] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351) >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at >>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331) >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79] -- Eric Evans john.eric.ev...@gmail.com
