I am running a middleware application in the tomcat environment described,
below. After rebooting the server, the memory consumption is couple of
gigs. Couple of weeks later, I get a message, I am out of memory.
Moreover, I need to bounce the whole server to start fresh.
Here, the log. I am quite sure how to go about troubleshooting it. Any
help is greatly appreciated.
catalina.out.prob:SEVERE: The web application [] appears to have started a
thread named [cluster-ClusterId{value='5745ebcecdb2e06579174645',
description='null'}-devnymongodb01.meridiancapital.com:27017] but has
failed to stop it. This is very likely to create a memory leak.
catalina.out.prob:SEVERE: Servlet.service() for servlet [API REST Handler]
in context with path [] threw exception [java.lang.OutOfMemoryError: unable
to create new native thread] with root cause
catalina.out.prob:java.lang.OutOfMemoryError: unable to create new native
thread
Here is my tomcat environment...
Server version: Apache Tomcat/7.0.69
Server built: Apr 11 2016 07:57:09 UTC
Server number: 7.0.69.0
OS Name: Linux
OS Version: 2.6.32-573.12.1.el6.x86_64
Architecture: amd64
JVM Version: 1.8.0_91-b14
JVM Vendor: Oracle Corporation
uname -a
Linux 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux
Mem info
MemTotal: 8061448 kB
MemFree: 5399052 kB
Buffers: 150360 kB
Cached: 388604 kB
SwapCached: 0 kB
Active: 2290720 kB
Inactive: 197764 kB
Active(anon): 1949532 kB
Inactive(anon): 160 kB
Active(file): 341188 kB
Inactive(file): 197604 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4128764 kB
SwapFree: 4128764 kB
Dirty: 40 kB
Writeback: 0 kB
AnonPages: 1949572 kB
Mapped: 35900 kB
Shmem: 176 kB
Slab: 87844 kB
SReclaimable: 27304 kB
SUnreclaim: 60540 kB
KernelStack: 5504 kB
PageTables: 9032 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8159488 kB
Committed_AS: 3091324 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 158244 kB
VmallocChunk: 34359576456 kB
HardwareCorrupted: 0 kB
AnonHugePages: 1767424 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 10240 kB
DirectMap2M: 8378368 kB
CPU info
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5649 @ 2.53GHz
stepping : 2
microcode : 29
cpu MHz : 2533.423
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc
aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt
aes hypervisor lahf_lm ida arat epb dts
bogomips : 5066.84
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5649 @ 2.53GHz
stepping : 2
microcode : 29
cpu MHz : 2533.423
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc
aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt
aes hypervisor lahf_lm ida arat epb dts
bogomips : 5066.84
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
--
Thanks,
R. Gelfand
