Hi all,
I am running into strange error when bootstrapping Cassandra cluster
in multiple datacenter setup.
The setup is as follows: 3 nodes in AWS east, 3 nodes somewhere on
Rackspace/Openstack. I use my own snitch based on EC2MultiRegionSnitch
(it just adds some ec2 avalability zone parsing capabilities). Nodes
in the cluster connect to each other and all seems ok.
When I start the Rackspace cluster first, populate it with data and
then let the AWS cluster bootstrap from it, it works great. However
the other way round it just breaks.
The breakage demonstrates as follows:
- nodetool rebuild us-east command hangs
- cassandra's log contains the following:
2013-03-19 12:42:15.796+0100 [Thread-14] [DEBUG]
IncomingTcpConnection.java(63)
org.apache.cassandra.net.IncomingTcpConnection: Connection version 6
from ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/xxx.xxx.xxx.xxx
2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
StreamInSession.java(104)
org.apache.cassandra.streaming.StreamInSession: Adding file
/path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
to Stream Request queue
2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
StreamInSession.java(104)
org.apache.cassandra.streaming.StreamInSession: Adding file
/path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-1-Data.db
to Stream Request queue
2013-03-19 12:42:15.806+0100 [Thread-14] [DEBUG]
IncomingStreamReader.java(112)
org.apache.cassandra.streaming.IncomingStreamReader: Receiving stream
2013-03-19 12:42:15.807+0100 [Thread-14] [DEBUG]
IncomingStreamReader.java(113)
org.apache.cassandra.streaming.IncomingStreamReader: Creating file for
/path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2-Data.db
with 7808 estimat
ed keys
2013-03-19 12:42:15.808+0100 [Thread-14] [DEBUG]
ColumnFamilyStore.java(863) org.apache.cassandra.db.ColumnFamilyStore:
component=key_space Checking for sstables overlapping []
2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-Data.db
2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-CompressionInfo.db
2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-TOC.txt
2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-Filter.db
2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-Index.db
2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] SSTable.java(154)
org.apache.cassandra.io.sstable.SSTable: Deleted
/path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2
2013-03-19 12:42:15.963+0100 [Thread-14] [INFO]
StreamInSession.java(136)
org.apache.cassandra.streaming.StreamInSession: Streaming of file
/path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
sections=127 progress=81048/2444
2013-03-19 12:42:16.059+0100 [Thread-13] [DEBUG]
IncomingTcpConnection.java(79)
org.apache.cassandra.net.IncomingTcpConnection: IOException reading
from socket; closing
java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
at
org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:93)
at
org.apache.cassandra.streaming.compress.CompressedInputStream.decompress(CompressedInputStream.java:101)
at
org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:79)
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
at
org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(BytesReadTracker.java:140)
at
org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:361)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
at
org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:160)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
at
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
2013-03-19 12:42:15.971+0100 [Thread-16] [ERROR]
CassandraDaemon.java(133)
org.apache.cassandra.service.CassandraDaemon: Exception in thread
Thread[Thread-16,5,main]
java.lang.RuntimeException: java.net.SocketException: Socket closed
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
at sun.security.ssl.InputRecord.read(InputRecord.java:509)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
at
org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 1 more
- the SSTable file is not downloaded.
To make it even more confusing, when I tried just one node in AWS and
one on Rackspace, the rebuild ran well. Also for two nodes. Only with
three on AWS side the process failed.
It's cassandra 1.2.1 running on
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.8) (rhel-1.56.1.11.8.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
SSL is configured for inter-DC communication.
Any hints?
regards,
Ondrej Cernos
