Hi again.
Now that I have (with your help) established some basic facts, I have a
practical case for analysis.
Before I get into details (because this is quite long), I'd like to
explain the reason why I'm asking this.
We have been testing a Tomcat application from a vendor. It works very
nicely when it is started on a test system, and does what we expect.
But the load it imposes on our test Tomcat at startup, makes me a bit
insecure about deploying it to a production server, before I really
understand what is happening.
Not being a Java or Tomcat expert, I'm just trying to collect some
pointers, to see if I can do something about what I am observing, or if
that's it and we just have to live with it.
The main thing is, I don't even know where to start, so I would
appreciate some help, even if very basic.
There is quite a bit of data below.
We have an old clunker PC (Pentium II, 512 Mb Ram, average ATA disks),
which we use to test some applications (not only Java/Tomcat).
Its a Linux system, and runs this Tomcat :
Using CATALINA_BASE: /usr/share/tomcat5.5
Using CATALINA_HOME: /usr/share/tomcat5.5
Using CATALINA_TMPDIR: /usr/share/tomcat5.5/temp
Using JRE_HOME: /usr/lib/jvm/java-1.5.0-sun
Server version: Apache Tomcat/5.5
Server built: Jan 20 2008 12:32:00
Server number: 5.5.20.0
OS Name: Linux
OS Version: 2.6.18-6-686
Architecture: i386
JVM Version: 1.5.0_14-b03
JVM Vendor: Sun Microsystems Inc.
The full command used to start Tomcat is :
/usr/lib/jvm/java-1.5.0-sun/bin/java
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties
-Djava.awt.headless=true -Xmx256M -server
-Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed -classpath
:/usr/lib/jvm/java-1.5.0-sun/jre//lib/jcert.jar:/usr/lib/jvm/java-1.5.0-sun/jre//lib/jnet.jar:/usr/lib/jvm/java-1.5.0-sun/jre//lib/jsse.jar:/usr/share/tomcat5.5/bin/bootstrap.jar:/usr/share/tomcat5.5/bin/commons-logging-api.jar
-Djava.security.manager
-Djava.security.policy==/var/lib/tomcat5.5/conf/catalina.policy
-Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5
-Djava.io.tmpdir=/var/lib/tomcat5.5/temp
org.apache.catalina.startup.Bootstrap start
Under Tomcat there, there is an application which we get from outside
and of which we do not have the source. This application is loaded at
Tomcat startup.
When Tomcat has been running for a while, but the system and Tomcat are
basically idle, a "top" with the option "n" (sort by memory usage),
shows this (first few lines only) :
top - 10:15:54 up 162 days, 16:25, 2 users, load average: 0.19, 0.09, 0.03
Tasks: 82 total, 1 running, 81 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.0%us, 0.3%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 516920k total, 509716k used, 7204k free, 71992k buffers
Swap: 1297564k total, 131340k used, 1166224k free, 95148k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6226 tomcat55 25 0 506m 191m 10m S 0.0 37.9 140:26.15 java
5937 root 22 0 267m 26m 8056 S 0.0 5.2 25:26.88 java
24608 root 23 0 260m 15m 9332 S 0.0 3.1 0:02.08 java
24638 root 15 0 29564 15m 5984 S 0.0 3.0 0:02.16 apache2
24627 efs 15 0 16964 12m 1968 S 0.0 2.6 0:06.68 MiraLoader.pl
24662 www-data 15 0 29564 10m 1632 S 0.0 2.2 0:02.91 apache2
24668 www-data 16 0 29564 10m 1624 S 0.0 2.2 0:02.50 apache2
24663 www-data 15 0 29564 10m 1620 S 0.0 2.2 0:02.78 apache2
etc..
(the other 2 java processes are separate java daemons, nothing to do
with Tomcat).
At this point Tomcat and this external application are very responsive,
at least in single-user mode. We have not really done yet a multi-user test.
Now, if I stop and restart Tomcat, for something like 6 minutes after
that, the same top display shows this kind of thing :
(There are 4 consecutive snapshots here, taken at more or less regular
intervals during the 6 minutes)
I)
top - 12:44:56 up 162 days, 18:54, 3 users, load average: 0.99, 0.27, 0.08
Tasks: 85 total, 3 running, 82 sleeping, 0 stopped, 0 zombie
Cpu(s): 98.0%us, 2.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 516920k total, 351160k used, 165760k free, 54548k buffers
Swap: 1297564k total, 70000k used, 1227564k free, 128152k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2864 tomcat55 25 0 463m 40m 11m R 97.9 7.9 0:25.62 java
5937 root 16 0 267m 24m 8056 S 0.0 4.9 25:38.84 java
24608 root 23 0 260m 15m 9332 S 0.0 3.1 0:02.17 java
24638 root 15 0 29564 15m 5984 S 0.0 3.0 0:02.16 apache2
24627 efs 15 0 16964 12m 1968 S 0.0 2.6 0:06.74 MiraLoader.pl
24664 www-data 15 0 29692 12m 3428 S 0.0 2.5 0:04.46 apache2
etc...
II)
top - 12:48:07 up 162 days, 18:57, 3 users, load average: 1.70, 0.96, 0.38
Tasks: 85 total, 2 running, 83 sleeping, 0 stopped, 0 zombie
Cpu(s): 95.0%us, 5.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 516920k total, 483396k used, 33524k free, 55680k buffers
Swap: 1297564k total, 70000k used, 1227564k free, 138644k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2864 tomcat55 25 0 461m 159m 12m R 99.2 31.5 3:30.88 java
5937 root 16 0 267m 24m 8056 S 0.0 4.9 25:38.84 java
24608 root 23 0 260m 15m 9332 S 0.0 3.1 0:02.17 java
24638 root 15 0 29564 15m 5984 S 0.0 3.0 0:02.16 apache2
24627 efs 15 0 16964 12m 1968 S 0.0 2.6 0:06.74 MiraLoader.pl
24664 www-data 15 0 29692 12m 3428 S 0.0 2.5 0:04.46 apache2
etc...
III)
top - 12:50:20 up 162 days, 19:00, 3 users, load average: 1.54, 1.15, 0.53
Tasks: 85 total, 2 running, 83 sleeping, 0 stopped, 0 zombie
Cpu(s): 94.4%us, 5.3%sy, 0.0%ni, 0.0%id, 0.3%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 516920k total, 509016k used, 7904k free, 51668k buffers
Swap: 1297564k total, 70000k used, 1227564k free, 142828k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2864 tomcat55 25 0 461m 184m 12m R 98.5 36.6 5:40.81 java
5937 root 16 0 267m 24m 8056 S 0.0 4.9 25:38.84 java
24608 root 23 0 260m 15m 9332 S 0.0 3.1 0:02.17 java
24638 root 15 0 29564 15m 5984 S 0.0 3.0 0:02.16 apache2
24627 efs 15 0 16964 12m 1968 S 0.0 2.6 0:06.74 MiraLoader.pl
24664 www-data 15 0 29692 12m 3428 S 0.0 2.5 0:04.46 apache2
etc...
IV) (Now it's back to idle, and Tomcat is responding)
top - 12:51:48 up 162 days, 19:01, 3 users, load average: 1.28, 1.19, 0.60
Tasks: 85 total, 1 running, 84 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.0%us, 0.3%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 516920k total, 490692k used, 26228k free, 43552k buffers
Swap: 1297564k total, 70000k used, 1227564k free, 130936k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2864 tomcat55 25 0 482m 188m 12m S 0.0 37.4 6:46.74 java
5937 root 16 0 267m 24m 8056 S 0.0 4.9 25:38.86 java
24608 root 23 0 260m 15m 9332 S 0.0 3.1 0:02.17 java
24638 root 15 0 29564 15m 5984 S 0.0 3.0 0:02.16 apache2
24627 efs 15 0 16964 12m 1968 S 0.0 2.6 0:06.75 MiraLoader.pl
24664 www-data 15 0 29692 12m 3428 S 0.0 2.5 0:04.46 apache2
etc...
In other words, while this application is being loaded, our Tomcat and
the whole system are totally unresponsive for about 5 minutes.
I know that it has to do with this application, for a number of reasons,
two of them being :
- if I do not start the application at Tomcat startup, Tomcat is
responsive after 15 seconds, instead of 5 minutes
- the Tomcat logfile shows me where things are not moving anymore during
those 5 minutes, and that is at the point of some message from this
application
- if I stop and restart this application separately with the Tomcat
Manager, I get the same 5 minute-freeze as when starting Tomcat
- I know also that this application has to read and parse a 25 Mb file
of XML data containing some data structure that it needs later on. I
don't know how it parses this, just that it does.
Now my questions are of the kind :
Our production servers are ASP servers, where several customers would be
using this same application, each customer with its own 25 Mb XML file
of data. They are more powerful servers, but if I have 5 customers and
these things compete with one another for memory or resources, I could
have a server that is unresponsive for a hour maybe at start.
So basically, I am asking if there are any parameters I can vary for the
application startup or Tomcat in general, to evaluate the behaviour
under different circumstances, or if I should put more RAM in the
server, or if I should lower or raise the amount of memory available to
Tomcat (and how), etc..
I am also a bit surprised about the amounts I see in terms of memory
usage, but not being an expert, I have not really anything to compare
them with. It's just that these java applications seem very hungry in
memory, if what is shown is really what they are using. For comparison,
we have other applications running on that server, of comparable
complexity, and they use 10-15 Mb apiece.
Anybody feel like commenting ?
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
