Analia,
Try running repair on node 3.
On Tue, May 12, 2015 at 7:39 AM, Analia Lorenzatto <
analialorenza...@gmail.com> wrote:
> Hello guys,
>
>
> I have a cluster 2.1.0-2 comprised of 3 nodes. The replication factor=2.
> We successfully added the third node last week. After that, We ran clean
> ups on one node at that time. Then we ran repairs on all the nodes, and
> finally compactions on all the CFs.
>
> Last night, I noticed the cluster started behaving in a weird way. The
> last node (successfully added last week) were being reported up and down
> all the time. I could see a lot of messages like this on logs:
>
> WARN [SharedPool-Worker-33] 2015-05-11 21:31:45,125
> AbstractTracingAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[SharedPool-Worker-33,5,main]: {}
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /mnt/cassandra/data/matchings-85b4929048e211e4a949a3ed319cbedc/matchings-ka-3914-Data.db
> (No such file or directory)
>
> At the same time the consumption of heap used was on the top, up to the
> point the rest of the cluster saw this node as down. After that, I just
> restarted the cassandra service with no problems on that node.
>
> Now, I can see the three nodes on the cluster Up and Normal, but this last
> node (which was rebooted) does not have data. But it has all the structure
> of cassandra data.
>
> I can query against the new node and I get the same result as if do the
> query against the others nodes. But, on this new node I do not have any
> SStables:
>
>
>
> root@prd-rtbkit-cassandra-03:/var/log/cassandra# nodetool status
> Datacenter: us-east
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.0.0.a 390.28 GB 256 66.7%
> eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c
> UN 10.0.0.b 382.36 GB 256 68.3%
> 19492c26-4458-4a0b-af04-72e0aab6598e 1c
> UN 10.0.0.c 40.61 MB 256 64.9%
> b8da952c-24b3-444a-a34e-7a1804eee6e6 1c
>
> What do you recommend to do? Leave this as if, remove it and try to join
> this or a new one?
> Thanks in advance!!
>
> --
> Saludos / Regards.
>
> Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life". By Captain Jean-Luc Picard.
>
--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick
Champion of Big Data (Cloudera)
http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html
2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
