On Tue, Sep 19, 2017 at 9:53 AM, Suvendu Sekhar Mondal <suv3...@gmail.com> wrote: > Radhika, > > On Tue, Sep 19, 2017 at 2:21 PM, Peddi, Radhika (Radhika) > <rpe...@avaya.com> wrote: >> Hi, >> >> We have upgraded RHEL 7.2 to RHEL 7.4. After upgrade when we are running >> performance testing of our application we are seeing below error in >> catalina.out. >> >> java.lang.OutOfMemoryError: unable to create new native thread Exception in >> Catalina.out after upgrading to RHEL 7.4
As stated before, this is a limitation on the number of threads spawned by the process imposed by the OS. In order to resolve it, you need to either increase nproc/nofile or decrease the number of threads that the Tomcat process is spawning. Check the ulimit for your user and compare that to the number of threads that you have configured. If you're on a 32-bit machine, it may be a problem with the process size, but RHEL-7 is only on 64-bit AFAIK so that shouldn't be your issue :) > > This simply indicates that JVM was trying to create a new thread and > OS can't create any new thread simply because native memory space was > exhausted. This limit is very much platform dependent. > > Is your app opening too many threads? On my Windows 10 laptop with > 32GB RAM and JDK 1.8, I can open 530000+ threads before run into this > OOM problem. You can find it by monitoring Tomcat instance. So far I > ran into this problem one time - couple of years ago. We got away > after adding some memory and by reducing thread stack size. > > Another thing can happen, you might have lost some OS setting(which > was used to bump up the default OS limit) while upgrading from one > version to another; which was preventing this issue from occurring so > far - just speculating. And this is not a Tomcat's problem. :) > >> Attached is the thread dump. >> >> We are using Tomcat 7.0.69. >> >> As per REDHAT we have increased tomcat5 user which used to run Tomcat >> process in /etc/security/limits.d/20-nproc.conf like below >> >> * soft nproc 4096 >> root soft nproc unlimited >> tomcat5 soft nproc 8192 >> >> All the processes are used by Tomcat. If we increase the value to 16K or 32K >> all threads are consumed and they are in Waiting(Parking) state. >> >> "http-bio-8443-exec-56" #973 daemon prio=5 os_prio=0 tid=0x00007efc8c029800 >> nid=0x10e7b waiting on condition [0x00007ef9bc7ee000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00000003c454c008> (a >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> at >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) >> at >> org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) >> at >> org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) >> at >> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at >> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >> at java.lang.Thread.run(Thread.java:745) >> >> "http-bio-8443-exec-55" #972 daemon prio=5 os_prio=0 tid=0x00007efc0401e000 >> nid=0x10e7a waiting on condition [0x00007ef9bc82f000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00000003c454c008> (a >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> at >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) >> at >> org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) >> at >> org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) >> at >> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at >> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >> at java.lang.Thread.run(Thread.java:745) >> >> They are not getting release. After some time Tomcat getting stopped >> automatically. You may need the output of below from our setup. >> >> [root@iqsvtk05<mailto:root@iqsvtk05> ~]# rpm -qif >> /opt/coreservices/tomcat-7.0.69/logs/ >> Name : tomcat >> Version : 7.0.69 >> Release : AV10 >> Architecture: x86_64 >> Install Date: Fri 18 Aug 2017 01:41:54 AM MDT >> Group : System Environment/Daemons >> Size : 8468465 >> License : Apache License, Version 2.0 >> Signature : (none) >> Source RPM : tomcat-7.0.69-AV10.src.rpm >> Build Date : Fri 24 Jun 2016 01:59:37 AM MDT >> Build Host : puiqx3650dev03.apac.avaya.com >> Relocations : /usr/share >> Packager : Avaya, Inc. >> Vendor : Apache Software Foundation >> URL : http://tomcat.apache.org/ >> Summary : Core Services Apache Jakarta Servlet/JSP server >> Description : Servlet container for developing Web applications in Java. >> Used as basis for the >> Reference Implementation of the Servlet and JSP specificiations. This RPM >> mirrors the layout used by official Tomcat distributions on all other OSes >> and linux variants. If you want a FHS-based package - be warned that tomcat >> may not work very well. (TODO: add more warnings !). If you still want >> FHS-based package - please use jpackage.org or an RPM that is using the same >> exact layout. >> >> Can you please help on this. >> >> Regards, >> Radhika > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
