It seems my system is running with an relatively old version of the Tomcat Native Library and old versions of APR and OpenSSL.
Tomcat Native 1.1.19 APR 1.3.8 OpenSSL 0.9.8k These are the latest versions available: Tomcat Native 1.1.23 APR 1.4.6 OpenSSL 1.0.1c I will try disabling the Tomcat Native Library to see if this solves the problem. Has anyone experience problems of Tomcat 7 running with these versions? -Jorge On Wed, Jun 6, 2012 at 2:33 PM, Jorge Medina <cerebrotecnolog...@gmail.com> wrote: > Hello, > I have an application that runs under Tomcat 7.0.23 that > periodically crashes. The java process running tomcat keeps growing in > memory until the Linux oom-killer kills the process. I do not get an > OutOfMemoryError because the memory leak is not in the Java heap. In > fact, it seems the heap is only using 4GB of the max 6GB specified in > the -Xmx parameter. Nevertheless the total memory held by the java > process keeps growing up to 16GB when the OS kills the process. > I haven't been able to find the conditions to reproduce this > problem, so I am not able to replicate it. Nevertheless it keeps > occurring; sometimes at midnight with no user activity sometimes in > the middle of a busy day. > > The web application uses Spring/Postgres/Mongo. > > I know this is not a Tomcat related problem, but some of you may > have experience a similar problem and may have some suggestions on how > to troubleshoot it. > I already have read many of the links that come after searching the > web for "java invoked oom-killer" but I still don't have any clue on > what causes the problem and how to solve it. > It looks like a memory leak in native code, not java code; so my > usual java toolset is not useful. > > Tomcat runs behind nginx in a EC2 instance. The application uses > Sun (now Oracle) JDK 1.6. > > Any suggestions on what should I look at? > > > -Jorge > > > Jun 4 16:02:49 ip-10-83-35-78 kernel: [1468800.179218] 3795110 pages > non-shared > Jun 5 06:50:07 ip-10-83-35-78 rsyslogd: [origin software="rsyslogd" > swVersion="4.2.0" x-pid="599" x-info="http://www.rsyslog.com";] > rsyslogd was HUPed, type 'lightweight'. > Jun 5 22:06:40 ip-10-83-35-78 kernel: [1576977.209487] java invoked > oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209492] java cpuset=/ > mems_allowed=0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209496] Pid: 15618, > comm: java Not tainted 2.6.32-317-ec2 #36-Ubuntu > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209498] Call Trace: > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209507] > [<ffffffff8107cbbc>] ? cpuset_print_task_mems_allowed+0x8c/0xc0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209513] > [<ffffffff810b1723>] oom_kill_process+0xe3/0x210 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209516] > [<ffffffff810b18a0>] __out_of_memory+0x50/0xb0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209519] > [<ffffffff810b195f>] out_of_memory+0x5f/0xc0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209523] > [<ffffffff810b4641>] __alloc_pages_slowpath+0x561/0x580 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209526] > [<ffffffff810b47d1>] __alloc_pages_nodemask+0x171/0x180 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209530] > [<ffffffff810b76f7>] __do_page_cache_readahead+0xd7/0x220 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209534] > [<ffffffff810b785c>] ra_submit+0x1c/0x20 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209536] > [<ffffffff810b01fe>] filemap_fault+0x3fe/0x450 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209541] > [<ffffffff810cbef0>] __do_fault+0x50/0x680 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209547] > [<ffffffff8102afdb>] ? __dequeue_entity+0x2b/0x50 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209550] > [<ffffffff810cde30>] handle_mm_fault+0x260/0x4f0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209555] > [<ffffffff814b3ab7>] do_page_fault+0x147/0x390 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209558] > [<ffffffff814b18e8>] page_fault+0x28/0x30 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209560] Mem-Info: > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209561] DMA per-cpu: > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209563] CPU 0: hi: > 0, btch: 1 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209565] CPU 1: hi: > 0, btch: 1 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209567] CPU 2: hi: > 0, btch: 1 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209569] CPU 3: hi: > 0, btch: 1 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209570] DMA32 per-cpu: > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209572] CPU 0: hi: > 155, btch: 38 usd: 44 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209574] CPU 1: hi: > 155, btch: 38 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209575] CPU 2: hi: > 155, btch: 38 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209577] CPU 3: hi: > 155, btch: 38 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209578] Normal per-cpu: > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209580] CPU 0: hi: > 155, btch: 38 usd: 32 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209582] CPU 1: hi: > 155, btch: 38 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209584] CPU 2: hi: > 155, btch: 38 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209585] CPU 3: hi: > 155, btch: 38 usd: 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209590] > active_anon:3513144 inactive_anon:266669 isolated_anon:0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209591] > active_file:101 inactive_file:15 isolated_file:0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209592] > unevictable:16 dirty:2 writeback:0 unstable:0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209593] free:19129 > slab_reclaimable:959 slab_unreclaimable:2729 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209594] mapped:0 > shmem:52 pagetables:0 bounce:0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209600] DMA > free:16384kB min:16kB low:20kB high:24kB active_anon:0kB > inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB > dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB > slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB > bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209604] > lowmem_reserve[]: 0 4024 15134 15134 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209611] DMA32 > free:48712kB min:4184kB low:5228kB high:6276kB active_anon:3632260kB > inactive_anon:24572kB active_file:224kB inactive_file:28kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:4120800kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB > shmem:4kB slab_reclaimable:472kB slab_unreclaimable:240kB > kernel_stack:80kB pagetables:0kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:107 all_unreclaimable? no > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209616] > lowmem_reserve[]: 0 0 11109 11109 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209623] Normal > free:11420kB min:11548kB low:14432kB high:17320kB > active_anon:10420316kB inactive_anon:1042104kB active_file:180kB > inactive_file:32kB unevictable:64kB isolated(anon):0kB > isolated(file):0kB present:11376528kB mlocked:64kB dirty:8kB > writeback:0kB mapped:0kB shmem:204kB slab_reclaimable:3364kB > slab_unreclaimable:10676kB kernel_stack:1960kB pagetables:0kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:99 > all_unreclaimable? no > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209628] > lowmem_reserve[]: 0 0 0 0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209631] DMA: 0*4kB > 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB > 4*4096kB = 16384kB > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209639] DMA32: 336*4kB > 194*8kB 141*16kB 97*32kB 73*64kB 57*128kB 28*256kB 18*512kB 6*1024kB > 1*2048kB 1*4096kB = 48896kB > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209646] Normal: > 953*4kB 23*8kB 11*16kB 14*32kB 12*64kB 11*128kB 8*256kB 2*512kB > 2*1024kB 0*2048kB 0*4096kB = 11916kB > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209653] 225 total > pagecache pages > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209654] 0 pages in swap cache > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209656] Swap cache > stats: add 0, delete 0, find 0/0 > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209657] Free swap = 0kB > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209658] Total swap = 0kB > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.252355] 3934208 pages RAM > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.252359] 118009 pages reserved > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.252361] 749 pages shared > Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.252362] 3795607 pages > non-shared > Jun 6 06:48:07 ip-10-83-35-78 rsyslogd: [origin software="rsyslogd" > swVersion="4.2.0" x-pid="599" x-info="http://www.rsyslog.com";] > rsyslogd was HUPed, type 'lightweight'. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
