Hi Aaron, I switched to 1.2.3 with no luck. I created https://issues.apache.org/jira/browse/CASSANDRA-5391 describing the problem. Maybe it's related to the EOFException problem, but I am not sure - I don't know Cassandra internals well and I have never seen the EOFException.
regards, ondrej On Tue, Mar 26, 2013 at 9:26 PM, aaron morton <aa...@thelastpickle.com> wrote: > If you are still on 1.2.1 may be this > > https://issues.apache.org/jira/browse/CASSANDRA-5105 > > Fixed in 1.2.2 > > If you are on 1.2.3 there is also > https://issues.apache.org/jira/browse/CASSANDRA-5381 > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 26/03/2013, at 5:10 AM, Ondřej Černoš <cern...@gmail.com> wrote: > > Hi all, > > I am still unable to move forward with this issue. > > - when I switch SSL off in inter-DC communication, nodetool rebuild works > well > - when I switch internode_compression off, I still get > java.io.IOException: FAILED_TO_UNCOMPRESS exception. Does > internode_compression: none really switch off the snappy compression > of the internode communication? The stacktrace - see the previous mail > - clearly demonstrates some compression is involved > - I managed to trigger another exception: > > java.lang.RuntimeException: javax.net.ssl.SSLException: bad record MAC > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) > at java.lang.Thread.run(Thread.java:662) > Caused by: javax.net.ssl.SSLException: bad record MAC > at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190) > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649) > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1607) > at > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:859) > at > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755) > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) > at > org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ... 1 more > > I managed to trigger this exception only once however. > > The fact the transfer works when SSL is off and fails with SSL is > another strange thing with this issue. > > Any ideas or hints? > > regards, > > Ondrej Cernos > > On Tue, Mar 19, 2013 at 5:51 PM, Ondřej Černoš <cern...@gmail.com> wrote: > > Hi all, > > I am running into strange error when bootstrapping Cassandra cluster > in multiple datacenter setup. > > The setup is as follows: 3 nodes in AWS east, 3 nodes somewhere on > Rackspace/Openstack. I use my own snitch based on EC2MultiRegionSnitch > (it just adds some ec2 avalability zone parsing capabilities). Nodes > in the cluster connect to each other and all seems ok. > > When I start the Rackspace cluster first, populate it with data and > then let the AWS cluster bootstrap from it, it works great. However > the other way round it just breaks. > > The breakage demonstrates as follows: > > - nodetool rebuild us-east command hangs > - cassandra's log contains the following: > > 2013-03-19 12:42:15.796+0100 [Thread-14] [DEBUG] > IncomingTcpConnection.java(63) > org.apache.cassandra.net.IncomingTcpConnection: Connection version 6 > from ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/xxx.xxx.xxx.xxx > 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG] > StreamInSession.java(104) > org.apache.cassandra.streaming.StreamInSession: Adding file > /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db > to Stream Request queue > 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG] > StreamInSession.java(104) > org.apache.cassandra.streaming.StreamInSession: Adding file > /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-1-Data.db > to Stream Request queue > 2013-03-19 12:42:15.806+0100 [Thread-14] [DEBUG] > IncomingStreamReader.java(112) > org.apache.cassandra.streaming.IncomingStreamReader: Receiving stream > 2013-03-19 12:42:15.807+0100 [Thread-14] [DEBUG] > IncomingStreamReader.java(113) > org.apache.cassandra.streaming.IncomingStreamReader: Creating file for > /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2-Data.db > with 7808 estimat > ed keys > 2013-03-19 12:42:15.808+0100 [Thread-14] [DEBUG] > ColumnFamilyStore.java(863) org.apache.cassandra.db.ColumnFamilyStore: > component=key_space Checking for sstables overlapping [] > 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110) > org.apache.cassandra.io.util.FileUtils: Deleting > key_space-column_family-tmp-ib-2-Data.db > 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110) > org.apache.cassandra.io.util.FileUtils: Deleting > key_space-column_family-tmp-ib-2-CompressionInfo.db > 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110) > org.apache.cassandra.io.util.FileUtils: Deleting > key_space-column_family-tmp-ib-2-TOC.txt > 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110) > org.apache.cassandra.io.util.FileUtils: Deleting > key_space-column_family-tmp-ib-2-Filter.db > 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] FileUtils.java(110) > org.apache.cassandra.io.util.FileUtils: Deleting > key_space-column_family-tmp-ib-2-Index.db > 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] SSTable.java(154) > org.apache.cassandra.io.sstable.SSTable: Deleted > /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2 > 2013-03-19 12:42:15.963+0100 [Thread-14] [INFO] > StreamInSession.java(136) > org.apache.cassandra.streaming.StreamInSession: Streaming of file > /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db > sections=127 progress=81048/2444 > 2013-03-19 12:42:16.059+0100 [Thread-13] [DEBUG] > IncomingTcpConnection.java(79) > org.apache.cassandra.net.IncomingTcpConnection: IOException reading > from socket; closing > java.io.IOException: FAILED_TO_UNCOMPRESS(5) > at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) > at org.xerial.snappy.SnappyNative.rawUncompress(Native Method) > at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391) > at > org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:93) > at > org.apache.cassandra.streaming.compress.CompressedInputStream.decompress(CompressedInputStream.java:101) > at > org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:79) > at > java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337) > at > org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(BytesReadTracker.java:140) > at > org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:361) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371) > at > org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:160) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226) > at > org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) > 2013-03-19 12:42:15.971+0100 [Thread-16] [ERROR] > CassandraDaemon.java(133) > org.apache.cassandra.service.CassandraDaemon: Exception in thread > Thread[Thread-16,5,main] > java.lang.RuntimeException: java.net.SocketException: Socket closed > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) > at java.lang.Thread.run(Thread.java:679) > Caused by: java.net.SocketException: Socket closed > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:146) > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554) > at sun.security.ssl.InputRecord.read(InputRecord.java:509) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850) > at > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:94) > at > org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ... 1 more > > - the SSTable file is not downloaded. > > To make it even more confusing, when I tried just one node in AWS and > one on Rackspace, the rebuild ran well. Also for two nodes. Only with > three on AWS side the process failed. > > It's cassandra 1.2.1 running on > > java version "1.6.0_24" > OpenJDK Runtime Environment (IcedTea6 1.11.8) > (rhel-1.56.1.11.8.el6_3-x86_64) > OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) > > SSL is configured for inter-DC communication. > > Any hints? > > regards, > > Ondrej Cernos > >
