-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
A C,
A C wrote:
> Application: Reporting application we'll call "reporting.war" , which
> unpacks to a directory size of ~ 100MB of jars/classes/static content/etc.
Aah, that's a different story. Certainly, not all 100MB is being loaded
into memory and staying there. You probably have images and stuff that
are never loaded.
> # webapp contexts (is that what these are called?): 40
The servlet spec and J2EE interchangeably call them "contexts" and "web
applications". I suppose a context only makes sense when the web
application is actually running. You could call a WAR file a "web
application", but you wouldn't call it a context. On the other hand, a
deployed web application is running in a context which .. well, IS the
web application.
Semantics.
> -Xms: 512MB
> -Xmx: 1024M
Just for kicks, set these to the same value.
Then, attach something to the Tomcat process that can read the heap info
(free, user, min/max) as well as request a full GC from the JVM. You can
use a profiler or something less bulky such as LambdaProbe.
Deploy a single web application (and leave the other 39 of them off).
Use the web app a bit to exercise it a little bit. Look at the memory
usage over time. You should get a mem usage curve that looks like this:
\/\/\/\/\/\/\/\/\/\
Nice ASCII art, eh?
Seriously, though, your memory should be allocated, then freed, pretty
regularly. You should be able to look at the peaks and valleys to
determine how much memory a single copy of your webapp requires while
running on Tomcat (yes, I know Tomcat is there too, but bear with me).
Next, deploy two instances of the webapp and repeat the process (make
sure to exercise both copies and not just one). You should be able to
see how much memory is used by two instances running on Tomcat. By
subtracting the amount you got from test 2 from test 1, you should be
able to guess at how much memory a single instance takes.
Try your guesses out by repeating the process with 3 or more instances
to see if the memory usage increases according to how much you think
each instance "takes".
It's possible that each instance needs maybe 50 or 60MB of heap space to
run comfortably. If you've capped the heap at 1GB, that means that you
can only have 1024 / 50 ~= 20 instances running comfortably in that
amount of memory (and that doesn't even cover the overhead of running
Tomcat itself). You either need to increase the available memory (you
can probably squeeze up to 1.6GB out of a 32-bit JVM or get yourself a
64-bit OS and JVM) or run some of these instances on another server.
> The only "thing" running on this machine is a Windows apache tomcat
> service. this application "hooks" into a sister app server that performs
> the "meat and potatos" activities of the application which we've never
> needed to restart but have done so to see if it was somehow related to
> web-layer OOMEs (hey, my father used to bang the t.v. set to make the
> picture more clear and it worked for HIM, so ...)
Do all 40 instances use that particular resource? What is it, may I ask?
> webapp/reporting/web-inf/lib = 93 jar files = 22 Mb
Note that memory footprint rarely has anything to do with the size of
the WAR file that was used to deploy it.
> Connection con = ds.getConnection;
>
> we do some SQL and then close/null any variables using or related to the
> connection/resultset/etc
Do you have this wrapped-up in a try/catch/finally block? Is it possible
that you have JDBC resource leaks?
You should set:
removeAbandoned="true"
removeAbandonedTimeout="30"
logAbandoned="true"
in your <Resource>. If your SQL queries take a long time to run, 30
seconds might not be enough "timeout" time, so feel free to increase that.
This configuration will log a stack trace for any connection that does
not get returned to the pool within (approximately)
"removeAbandonedTimeout" seconds. You might find that you are leaking
connections. Then again, you might not be. It's always good to check.
> Here's the permgen after 1.5 hours of my simulation (35 deployed webapps of
> this application)
Let's see how your heap looks after a while...
> 1247.073: [GC 1247.073: [DefNew: 51055K->276K(52480K), 0.0630231 secs]
> 123930K->75721K(518528K), 0.0630960 secs]
You are reclaiming a bunch of memory in each GC.
Let's keep going:
> 1538.764: [Full GC 1538.764: [Tenured: 86527K->59881K(466048K),
> 5.3928429secs] 133369K->59881K(518528K), [Perm :
> 32768K->32610K(32768K)], 5.3929301 secs]
Looks like you're busting PermGen. You nearly busted it at the last Full
GC (had 32768K in it, and freed only about 150K).
> 2357.129: [Full GC 2357.129: [Tenured: 89508K->71494K(466048K),
> 7.1956506secs] 105965K->71494K(518528K), [Perm :
> 49151K->48973K(49152K)], 7.2276270 secs]
PermGen size has increased, but you're still very close to filling it up.
Note that the full heap is still only 1/2 GB (your initial heap size). 7
seconds is a loooong GC.
> 2485.690: [Full GC 2485.690: [Tenured: 90945K->91468K(466048K),
> 3.2931426secs] 138286K->91468K(518528K), [Perm :
> 53247K->53247K(53248K)],
PermGen keeps getting bigger. Over what time period was this?
> 2969.556: [Full GC 2969.556: [Tenured: 97335K->97396K(466048K),
> 0.4763756secs] 97664K->97396K(518528K), [Perm :
> 65535K->65535K(65536K)],
> 0.4764480 secs]
Does it bomb after this? You are always tight on PermGen space. I'd
guess this was the problem.
> when i remove some 30-odd *.xml deployment files from /conf.../localhost and
> have it load just TWO webapps, it lasts much longer
Well, whatever the problem is, having 40 instances is certainly
compounding it. Does it eventually die when running only two webapps?
> what should i do? is this a PermGen problem?
Almost certainly. Take a look at the rest of your heap: you are never
really exceeding your initial heap size, so it's not regular heap space
you need... it's PermGen.
> And if it is, why does my
> error message keep saying "ServeletException" and not the OutOfMemory
> PermGen message?
It doesn't always say "PermGen ran out of space".
> thanks - and if anyone needs graphs/files/blood i'll be happy to drop them
> somewhere
Didn't there used to be some kind of UNIX administrators guide to
sacrifices and stuff? I remember reading about what you were supposed to
do with chicken blood when you were having some specific issue waaaay
back in the day. Who knows; maybe it'll help.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHD9oC9CaO5/Lv0PARAq8HAJ4+EjFXkVKgkdFp4s8unGlIsj/nWwCfWvW6
+ZGCvQhfywUKuFIJOjf4kWc=
=FJyG
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
