Lennart Liberg created CASSANDRA-8628:
-----------------------------------------
Summary: Stalling rebuild of new 2.1.12 node from cluster running
2.0.9
Key: CASSANDRA-8628
URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
Cluster running cassandra 2.0.9, new node running 2.1.2,
inter-node encryption: all
Reporter: Lennart Liberg
We wanted to add a new datacenter to our cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the system.log output is observed, with
a new UnknownColumnFamilyException and NullPointerException every few minutes,
and no data being transfered.
After downgrading to cassandra 2.0.11, the rebuild operation worked flawless,
using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
file (a lot of new items in 2.1.x).
My reading of the upgrading compatibility document does not lead me to think
there does not ought to be any problem in performing this kind of rebuild.
http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
system.log output:
{noformat}
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
StorageService.java:958 - rebuild from dc: ams1
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Executing streaming plan for Rebuild
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Starting streaming to 37.xx.xx.1/37.xx.xx.1
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=d1ea009c-feba-3d75-aa64-d022270842c4
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-28974,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
