> I used JMX to check current number of threads in a production cassandra > machine, and it was ~27,000. That does not sound too good.
My first guess would be lots of client connections. What client are you using, does it do connection pooling ? See the comments in cassandra.yaml around rpc_server_type, the default uses sync uses one thread per connection, you may be better with HSHA. But if your app is leaking connection you should probably deal with that first. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 30/04/2013, at 3:07 AM, William Oberman <ober...@civicscience.com> wrote: > Hi, > > I'm having some issues. I keep getting: > ------------ > ERROR [GossipStage:1] 2013-04-28 07:48:48,876 AbstractCassandraDaemon.java > (line 135) Exception in thread Thread[GossipStage:1,5,main] > java.lang.OutOfMemoryError: unable to create new native thread > -------------- > after a day or two of runtime. I've checked and my system settings seem > acceptable: > memlock=unlimited > nofiles=100000 > nproc=122944 > > I've messed with heap sizes from 6-12GB (15 physical, m1.xlarge in AWS), and > I keep OOM'ing with the above error. > > I've found some (what seem to me) to be obscure references to the stack size > interacting with # of threads. If I'm understanding it correctly, to reason > about Java mem usage I have to think of OS + Heap as being locked down, and > the stack gets the "leftovers" of physical memory and each thread gets a > stack. > > For me, the system ulimit setting on stack is 10240k (no idea if java sees or > respects this setting). My -Xss for cassandra is the default (I hope, don't > remember messing with it) of 180k. I used JMX to check current number of > threads in a production cassandra machine, and it was ~27,000. Is that a > normal thread count? Could my OOM be related to stack + number of threads, > or am I overlooking something more simple? > > will >
