Looks like https://issues.apache.org/jira/browse/CASSANDRA-5381 can you add your experience to the ticket ?
Thanks ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 25/03/2013, at 10:45 PM, Mathijs Vogelzang <math...@apptornado.com> wrote: > Hi all, > > We have a 6-node cassandra cluster which has worked fine for a long > time through upgrades starting from 0.8.x to 1.1.x. Recently we > upgraded to 1.2.2, and since then streaming repair doesn't work > anymore (everything else works, gossip, serving Thrift queries etc.). > We upgraded to 1.2.3, upgraded the JDK to the latest version (1.7u17), > but nothing helped. The only error message in the logs is the > following pasted below: > > INFO [AntiEntropyStage:1] 2013-03-25 09:30:33,493 > StreamOutSession.java (line 162) Streaming to /xxx.xxx.xxx.xxx > INFO [Streaming to /10.181.129.193:1] 2013-03-25 09:30:33,859 > StreamReplyVerbHandler.java (line 50) Need to re-stream file > /var/lib/cassandra/data/....db to /xxx.xxx.xxx.xxx > INFO [Streaming to /10.181.129.193:1] 2013-03-25 09:30:33,994 > StreamReplyVerbHandler.java (line 50) Need to re-stream file > /var/lib/cassandra/data/....db to /xxx.xxx.xxx.xxx > INFO [Streaming to /10.181.129.193:1] 2013-03-25 09:30:34,190 > StreamReplyVerbHandler.java (line 50) Need to re-stream file > /var/lib/cassandra/data/.....db to /xxx.xxx.xxx.xxx > ERROR [Streaming to /10.181.129.193:1] 2013-03-25 09:30:34,474 > CassandraDaemon.java (line 164) Exception in thread Thread[Streaming > to /xxx.xxx.xxx.xxx:1,5,main] > java.lang.RuntimeException: java.io.EOFException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:392) > at > org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) > at > org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) > at > org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ... 3 more > > Subsequently the repair command hangs, and the nodes start running out > of memory after a few cycles with the heap being full of Merkle tree > related datastructures. > > We've now discovered that when we turn internode encryption off then > the streaming works again. Is there something that could explain why > the regular internode network traffic works (else thrift queries > should also fail), but the streaming doesn't? > > Our internode encryption settings were: > server_encryption_options: > internode_encryption: all > keystore: conf/.keystore > keystore_password: xxxxxxxx > truststore: conf/.truststore > truststore_password: xxxxxxxx > protocol: TLS > algorithm: SunX509 > store_type: JKS > cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA] > > > Best regards, > > Mathijs
