I'm finding it hard to believe, but all points that the problem was the -Xms option of the Oracle (Sun) JVM. I originally set it to the same value as -Xmx, so that all memory for the heap is allocated when the JVM starts. This works fine in Solaris, but it is not working in Ubuntu. After removing that option, the JVM process memory usage seems to keep stable.
I am using Sun JVM 1.6.0.26 -Jorge On Thu, Jun 7, 2012 at 11:38 PM, Pid * <p...@pidster.com> wrote: > On 7 Jun 2012, at 23:03, Daniel Mikusa <dmik...@vmware.com> wrote: > >> ----- Original Message ----- >>> Only 52 java threads. It used to fluctuate more (we made some >>> changes >>> to the app to perform a task in a single thread rather than spawning >>> multiple threads, but the crash still occurs) . The number of threads >>> is always below 100. >>> >>> jstack -F 21370 | grep ^Thread | wc -l >>> ps -T -p 21370 (This gives me 63) >>> >>> I don't seem to specify the -Xss option: >> >> In some applications with a large number of threads (particularly when >> running on 64-bit hardware) this setting can cause a problems. The default >> value is pretty large (I think it's 1M on 64-bit systems). Since most apps >> don't need that large of a value, an easy performance tuning step is to >> lower the value of -Xss. >> >> Since you don't have very many threads, it seems unlikely that this is >> causing your problem though. >> >> That being said, you could try explicitly setting a value for the thread >> stack size. Finding the right values takes some testing though. I usually >> start with something like 192k and run a few application tests. If I see >> any stack overflow exceptions then I increase the value and rerun the tests. >> Repeat until there are no stack overflow exceptions. >> >> >> On a different note, what is the specific version of the JVM that you are >> running? If it's not the latest, you could always try upgrading to the >> latest version. > > You need to hook up the VisualVM + Memory Pools plugin. > > This will show you where the memory is being consumed, if it's by the JVM. > > > p > > > >> >> >>> >>> Xms6g -Xmx6g -XX:NewSize=4G -XX:MaxNewSize=4G -XX:SurvivorRatio=6 >>> -XX:MaxPermSize=512M -XX:-UseConcMarkSweepGC -XX:+UseStringCache >>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/example/logs >>> >>> -Jorge >>> >>> >>> >>> >>> >>> >>> On Thu, Jun 7, 2012 at 12:07 PM, Daniel Mikusa <dmik...@vmware.com> >>> wrote: >>>> ----- Original Message ----- >>>>> I am using MongoDB through the Java driver allowing up to 100 >>>>> connections to the MongoDB server. >>>>> I also use DBCP with a max size of 50 JDBC connections. >>>>> My webapp uses about 150 JAR files. >>>>> There is no native libraries loaded from my webapp as far as I >>>>> know. >>>>> All the app is pure Java code. (Nevertheless, Tomcat is using the >>>>> Tomcat Native Library) >>>>> >>>>> Is there a way I can monitor the number of file descriptors in use >>>>> by >>>>> the app? >>>>> >>>>> I have monitored the number of threads, but I haven't seen >>>>> anything >>>>> unusual. >>>> >>>> How many threads have you observed? Total threads, not just >>>> threads for the connector. >>>> >>>> Also, what is the value you are using for thread stack size? -Xss >>>> >>>> Dan >>>> >>>> >>>> >>>> (but it could be that the burst is too fast to get catch by >>>>> the monitoring tool) >>>>> >>>>> -Jorge >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Jun 7, 2012 at 11:44 AM, Christopher Schultz >>>>> <ch...@christopherschultz.net> wrote: >>>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>>> Hash: SHA1 >>>>>> >>>>>> Jorge, >>>>>> >>>>>> On 6/6/12 5:33 PM, Jorge Medina wrote: >>>>>>> The web application uses Spring/Postgres/Mongo. >>>>>> >>>>>> Are you using MongoDB in-process or anything weird like that? Or >>>>>> are >>>>>> you connecting through some socket-based (or other) API? >>>>>> >>>>>>> It looks like a memory leak in native code, not java code; so >>>>>>> my >>>>>>> usual java toolset is not useful. >>>>>> >>>>>> If what you are observing is accurate (non-heap memory grows, >>>>>> heap >>>>>> stays reasonable) then it will definitely be more difficult to >>>>>> track-down. >>>>>> >>>>>>> Tomcat runs behind nginx in a EC2 instance. The application >>>>>>> uses >>>>>>> Sun (now Oracle) JDK 1.6. >>>>>>> >>>>>>> Any suggestions on what should I look at? >>>>>> >>>>>> What do your <Connectors> look like? How many JDBC connections >>>>>> do >>>>>> you >>>>>> have in your connection pool (which you are hopefully using!)? >>>>>> How >>>>>> about the same equivalent for MongoDB? >>>>>> >>>>>> Does your webapp keep lots of files open? Do you have an >>>>>> unusually-large number of JAR files in your webapp? Do you have >>>>>> any >>>>>> native libraries in use within your webapp? >>>>>> >>>>>> What are all the non-default system properties that you are >>>>>> setting >>>>>> at >>>>>> JVM launch time (you can easily see this from a 'ps' list)? >>>>>> >>>>>> Two things that can eat-up native memory fast in a JVM are file >>>>>> descriptors and threads, so let's start there. >>>>>> >>>>>> - -chris >>>>>> -----BEGIN PGP SIGNATURE----- >>>>>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin) >>>>>> Comment: GPGTools - http://gpgtools.org >>>>>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ >>>>>> >>>>>> iEYEARECAAYFAk/Q9ooACgkQ9CaO5/Lv0PDPyQCfVtddxMDOgQbjmMGC3gvnK+Qq >>>>>> aZMAnjVu67+9Sm2bdYzAd91ZOrYo3DFI >>>>>> =r+vl >>>>>> -----END PGP SIGNATURE----- >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
