Hello guys,
I have a cluster 2.1.0-2 comprised of 3 nodes. The replication factor=2.
We successfully added the third node last week. After that, We ran clean
ups on one node at that time. Then we ran repairs on all the nodes, and
finally compactions on all the CFs.
Last night, I noticed the cluster started behaving in a weird way. The
last node (successfully added last week) were being reported up and down
all the time. I could see a lot of messages like this on logs:
WARN [SharedPool-Worker-33] 2015-05-11 21:31:45,125
AbstractTracingAwareExecutorService.java:167 - Uncaught exception on thread
Thread[SharedPool-Worker-33,5,main]: {}
java.lang.RuntimeException: java.io.FileNotFoundException:
/mnt/cassandra/data/matchings-85b4929048e211e4a949a3ed319cbedc/matchings-ka-3914-Data.db
(No such file or directory)
At the same time the consumption of heap used was on the top, up to the
point the rest of the cluster saw this node as down. After that, I just
restarted the cassandra service with no problems on that node.
Now, I can see the three nodes on the cluster Up and Normal, but this last
node (which was rebooted) does not have data. But it has all the structure
of cassandra data.
I can query against the new node and I get the same result as if do the
query against the others nodes. But, on this new node I do not have any
SStables:
root@prd-rtbkit-cassandra-03:/var/log/cassandra# nodetool status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.0.0.a 390.28 GB 256 66.7%
eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c
UN 10.0.0.b 382.36 GB 256 68.3%
19492c26-4458-4a0b-af04-72e0aab6598e 1c
UN 10.0.0.c 40.61 MB 256 64.9%
b8da952c-24b3-444a-a34e-7a1804eee6e6 1c
What do you recommend to do? Leave this as if, remove it and try to join
this or a new one?
Thanks in advance!!
--
Saludos / Regards.
Analía Lorenzatto.
“It's possible to commit no errors and still lose. That is not weakness.
That is life". By Captain Jean-Luc Picard.
