For start: - check (cassandra-env.sh) -Xss size, you may need to increase it for your JVM; - check (cassandra-env.sh) -Xms and -Xmx size, you may need to increase it for your data load/bloom filter/index sizes.
Best regards / Pagarbiai Viktor Jevdokimov Senior Developer [Adform News] <http://www.adform.com> Visit us at Dmexco: Hall 6 Stand B-52 September 18-19 Cologne, Germany Email: viktor.jevdoki...@adform.com<mailto:viktor.jevdoki...@adform.com> Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-03163 Vilnius, Lithuania Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider> Take a ride with Adform's Rich Media Suite<http://vimeo.com/adform/richmedia> [Dmexco 2013] <http://www.dmexco.de/> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: srmore [mailto:comom...@gmail.com] Sent: Tuesday, September 10, 2013 6:16 AM To: user@cassandra.apache.org Subject: Error during startup - java.lang.OutOfMemoryError: unable to create new native thread [heur] I have a 5 node cluster with a load of around 300GB each. A node went down and does not come up. I can see the following exception in the logs. ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[main,5,main] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:640) at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703) at java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1392) at org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:77) at org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:65) at org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor.<init>(JMXConfigurableThreadPoolExecutor.java:34) at org.apache.cassandra.concurrent.StageManager.multiThreadedConfigurableStage(StageManager.java:68) at org.apache.cassandra.concurrent.StageManager.<clinit>(StageManager.java:42) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:344) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:173) The ulimit -u output is 515042 Which is far more than what is recommended [1] (10240) and I am skeptical to set it to unlimited as recommended here [2] Any pointers as to what could be the issue and how to get the node up. [1] http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=install/recommended_settings#cassandra/install/installRecommendSettings.html [2] http://mail-archives.apache.org/mod_mbox/cassandra-user/201303.mbox/%3CCAPqEvGE474Omea1BFLJ6U_pbAkOwWxk=dwo35_pc-atwb4_...@mail.gmail.com%3E Thanks !
<<inline: signature-logo734.png>>
<<inline: dmexco4bc1.png>>
