Hi, I'm having some issues. I keep getting: ------------ ERROR [GossipStage:1] 2013-04-28 07:48:48,876 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[GossipStage:1,5,main] java.lang.OutOfMemoryError: unable to create new native thread -------------- after a day or two of runtime. I've checked and my system settings seem acceptable: memlock=unlimited nofiles=100000 nproc=122944
I've messed with heap sizes from 6-12GB (15 physical, m1.xlarge in AWS), and I keep OOM'ing with the above error. I've found some (what seem to me) to be obscure references to the stack size interacting with # of threads. If I'm understanding it correctly, to reason about Java mem usage I have to think of OS + Heap as being locked down, and the stack gets the "leftovers" of physical memory and each thread gets a stack. For me, the system ulimit setting on stack is 10240k (no idea if java sees or respects this setting). My -Xss for cassandra is the default (I hope, don't remember messing with it) of 180k. I used JMX to check current number of threads in a production cassandra machine, and it was ~27,000. Is that a normal thread count? Could my OOM be related to stack + number of threads, or am I overlooking something more simple? will
