Hi,
From yesterday, I'm trying to add a new node to an existing 3 nodes
Cassandra cluster, running version 1.2.3. Today I have started clean,
removed the node that I have tried to add yesterday and try to add
another node. Verified both cassandra.yaml and
cassandra-topology.properties are identical on all nodes (I'm using
NetworkTopologyStrategy and PropertyFile snitch).
I see the nodetool netstats on the new node shows that it is streaming
from the other nodes, but other than increasing number on Responses
Pool, there is no change in the report. Logs are set to Debug and most
of new lines are "Applying mutations".
Below is the only ERROR that I can find in both cassandra.log and
system.log. So, it seems that streaming is working but not
completing... For the sstable that I get the error, the percentage is
not increasing in netstats, not for any other sstables. Now it is
running for few hours and no progress. Any clue ?
ERROR 06:09:03,934 Exception in thread Thread[Thread-25,5,main]
java.lang.AssertionError: incorrect row data size 114969 written to
/data/woow_backend/offline/woow_backend-offline-tmp-ib-2-Data.db;
correct is 115006
at
org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
at
org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
at
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
nodetool netstats:
Mode: JOINING
Not sending any streams.
Streaming from: /10.11.1.108
woow_backend:
/data/woow_backend/user_rooms/woow_backend-user_rooms-ib-1-Data.db
sections=1 progress=0/566 - 0%
woow_backend:
/data/woow_backend/gc_archive/woow_backend-gc_archive-ib-2-Data.db
sections=1 progress=0/3478 - 0%
woow_backend:
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-258-Data.db
sections=1 progress=0/313 - 0%
woow_backend:
/data/woow_backend/roster_groups/woow_backend-roster_groups-ib-23-Data.db sections=1
progress=0/120 - 0%
woow_backend:
/data/woow_backend/roster_versions/woow_backend-roster_versions-ib-34-Data.db
sections=2 progress=0/195 - 0%
woow_backend:
/data/woow_backend/rosters/woow_backend-rosters-ib-27-Data.db
sections=93 progress=0/881196 - 0%
woow_backend:
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-257-Data.db
sections=104 progress=0/5816889 - 0%
woow_backend:
/data/woow_backend/avatars/woow_backend-avatars-ib-68-Data.db
sections=71 progress=0/5841975 - 0%
woow_backend:
/data/woow_backend/rosters/woow_backend-rosters-ib-28-Data.db sections=2
progress=0/234 - 0%
woow_backend:
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-259-Data.db
sections=3 progress=0/6910 - 0%
woow_backend:
/data/woow_backend/offline/woow_backend-offline-ib-129-Data.db
sections=77 progress=22499/462727 - 4%
woow_backend:
/data/woow_backend/rooms/woow_backend-rooms-ib-2-Data.db sections=1
progress=0/308 - 0%
woow_backend:
/data/woow_backend/roster_versions/woow_backend-roster_versions-ib-33-Data.db
sections=93 progress=0/34246 - 0%
woow_backend:
/data/woow_backend/rooms/woow_backend-rooms-ib-1-Data.db sections=1
progress=0/310 - 0%
Pool Name Active Pending Completed
Commands n/a 0 15
Responses n/a 0 44808
Thanks,
Gabi
