I have a 5 node cluster with a load of around 300GB each. A node went down
and does not come up. I can see the following exception in the logs.
ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line
139) Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
at
java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1392)
at
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:77)
at
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:65)
at
org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor.<init>(JMXConfigurableThreadPoolExecutor.java:34)
at
org.apache.cassandra.concurrent.StageManager.multiThreadedConfigurableStage(StageManager.java:68)
at
org.apache.cassandra.concurrent.StageManager.<clinit>(StageManager.java:42)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:344)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:173)
The *ulimit -u* output is
*515042*
Which is far more than what is recommended [1] (10240) and I am skeptical
to set it to unlimited as recommended here [2]
Any pointers as to what could be the issue and how to get the node up.
[1]
http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=install/recommended_settings#cassandra/install/installRecommendSettings.html
[2]
http://mail-archives.apache.org/mod_mbox/cassandra-user/201303.mbox/%3CCAPqEvGE474Omea1BFLJ6U_pbAkOwWxk=dwo35_pc-atwb4_...@mail.gmail.com%3E
Thanks !
