Re: Cassandra node going down

2012-09-14 Thread Tyler Hobbs
You will need to run nodetool removetoken with the old node's token to permanently remove it from the cluster. On Fri, Sep 14, 2012 at 3:06 PM, rohit reddy wrote: > Thanks for the inputs. > The disk on the EC2 node failed. This led to the problem. Now i have > created a new cassandra node and add

Re: Cassandra node going down

2012-09-14 Thread rohit reddy
Thanks for the inputs. The disk on the EC2 node failed. This led to the problem. Now i have created a new cassandra node and added it to the cluster. Do i need to do anything to delete the old node from the cluster, or will the cluster balance it self. Asking this since in Datastax ops center its

Re: Cassandra node going down

2012-09-14 Thread Robin Verlangen
Cassandra writes to memtables, that will get flushed to disk when it's time. That might be because of running out of memory (the log message you just posted), on a shutdown, or at other times. That's why you're using memory while writing. You seem to be running on AWS, are you sure your data locat

Re: Cassandra node going down

2012-09-14 Thread rohit reddy
Hi Robin, I had checked that. Our disk size is about 800GB, and the total data size is not more than 40GB. Even if all the data is stored in one node, this won't happen. I'll try to see if the disk failed. Is this anything to do with VM memory?.. cause this logs suggests that.. Heap is 0.7515559

Re: Cassandra node going down

2012-09-14 Thread Robin Verlangen
Robbit = Rohit of course, excuse me. Best regards, Robin Verlangen *Software engineer* * * W http://www.robinverlangen.nl E ro...@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidenti

Re: Cassandra node going down

2012-09-14 Thread Robin Verlangen
Hi Robbit, I think it's running out of disk space, please verify that (on Linux: df -h ). Best regards, Robin Verlangen *Software engineer* * * W http://www.robinverlangen.nl E ro...@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attentio

Cassandra node going down

2012-09-14 Thread rohit reddy
Hi, I'm facing a problem in Cassandra cluster deployed on EC2 where the node is going down under write load. I have configured a cluster of 4 Large EC2 nodes with RF of 2. All nodes are instance storage backed. DISK is RAID0 with 800GB I'm pumping in write requests at about 4000 writes/sec. One