Re: major hdfs issues

2011-03-12 Thread Todd Lipcon
You may also want to look at the value set for ulimit -u - it's unlimited on many OSes, but RHEL6 in particular sets it way too low, which will cause the "unable to creative native thread". What OS you running? The conntrack error has to do with ip_conntrack, which is an iptables module that keeps

Re: major hdfs issues

2011-03-12 Thread Jack Levin
Awesome, thanks... This is similar to mysql max-conn setting. -Jack On Sat, Mar 12, 2011 at 11:29 AM, Stack wrote: > I opened HBASE-3628 to expose the TThreadPoolServer options on the > command-line for thrift server. > St.Ack > > On Sat, Mar 12, 2011 at 11:20 AM, Stack wrote: > > Via Bryan (a

Re: major hdfs issues

2011-03-12 Thread Stack
I opened HBASE-3628 to expose the TThreadPoolServer options on the command-line for thrift server. St.Ack On Sat, Mar 12, 2011 at 11:20 AM, Stack wrote: > Via Bryan (and J-D), by default we use the thread pool server from > Thrift (unless you choose the non-blocking option): > > 978       LOG.inf

Re: major hdfs issues

2011-03-12 Thread Stack
Via Bryan (and J-D), by default we use the thread pool server from Thrift (unless you choose the non-blocking option): 978 LOG.info("starting HBase ThreadPool Thrift server on " + listenAddress + ":" + Integer.toString(listenPort)); 979 server = new TThreadPoolServer(processor, serverT

Re: major hdfs issues

2011-03-12 Thread Stack
I don't see any bounding in the thrift code. Asking Bryan St.Ack On Sat, Mar 12, 2011 at 10:04 AM, Jack Levin wrote: > So our problem is this: when we restart a region server, or it goes > down, hbase slows down, while we send super high frequency thrift > calls from our PHP front-end APP we

Re: major hdfs issues

2011-03-12 Thread Jack Levin
So our problem is this: when we restart a region server, or it goes down, hbase slows down, while we send super high frequency thrift calls from our PHP front-end APP we actually spawn up 2+ threads on thrift, and what this does is destroys all memory on the boxes, and causes DNs just to shut d

Re: major hdfs issues

2011-03-12 Thread Suraj Varma
>> to:java.lang.OutOfMemoryError: unable to create new native thread This indicates that you are oversubscribed on your RAM to the extent that the JVM doesn't have any space to create native threads (which are allocated outside of the JVM heap.) You may actually have to _reduce_ your heap sizes t