Ok, I think I nailed it.
There is a memory leak in Tomcat 5. I downloaded a trial of JProbe (they
should provide free licenses for open source! :)
There is a class called o.a.coyote.RequestGroupInfo.java, in that class
there is a method called addRequestProcessor.
This method stores a reference to a RequestInfo object in an ArrayList, the
RequestInfo object has a reference to a Request object and that eventually
holds the reference to a MessageBytes/ByteChunk.
The nr of instances of ByteChunk and CharChunk keep growing and they never
get garbage collected.
To validate my thoughts, I commented out the line
public void addRequestProcessor( RequestInfo rp ) {
//processor.add( rp );
}
in RequestGroupInfo.java and now the instance count for ByteChunk stay
constant.
I have a feeling that the actual solution is a little bit deeper down, right
now I am eager on getting the clustering to perform the way I want, so if
someone wants to take this on be my guest and let me know. If I don't hear
from anyone, I will commit a solution to this next week.
Filip
-----Original Message-----
From: Filip Hanik [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 5:22 PM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
RE: Memory leak- yeah I knowhi Michael, thanks for your info. I will look
into that as well and set those settings
The thing I am investigating right now is if the fact that we are
registering a lot of things with JMX causes some references to never be
released.
more info to follow.
Filip
-----Original Message-----
From: Michael Yates [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 4:43 PM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
Filip,
I spent some time a while back trying to get some reproducibility into
memory use in Tomcat (4.1.24)
I found that even sitting idle tomcats memory use would increase.
Here is my original mail to the tomcat-dev mailing list
<<4.1.24 secure connector throws a LOT of garbage>>
After I did some more investigations and figured out a fix the following
bug was raised
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=19264
To reduce the amount of garbage thrown by tomcat I would recommend two
things, doing these might help you track down other memory issues.
1 - Use -Xmx<someValue>m -Xms<someValue>m.
The above sizes the heap to whatever value you put in there (note the
value is megabytes). I found (at least on the Solaris JVM) that sizing the
heap and making the min and max heap sizes the same not only sped up start
up but also made the garbage collector behave much better.
2- Set the serverSoTimeout value to 0 in the connector configuration in
your server configuration file. This will prevent the sever socket timing
out. Which DOES throw garbage, and if it is a secure connector you are using
it throws SO much garbage that the garbage collector can not keep up too
well.
Using just number tip 1 above (as the fix to the bug above wasn't in
Tomcat 4.1.x before we needed to go into production. We managed to run at
something like 100 TPS for 48 hours with no falling over and no net increase
in memory usage.
If you need any more information on the above let me know.
Michael Yates
Software Engineer, Location Center
Nortel Networks Wollongong, Australia.
-----Original Message-----
From: Filip Hanik [mailto:[EMAIL PROTECTED]
Sent: Thursday, 15 January 2004 11:09 AM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
Remy, absolutely no intentions of being sarcastic :), if there isn't a
leak, that is great, if there is, will fix it.
I'm trying to figure it out, I didn't ask anyone else to spend time on it,
or even read my email. but if anyone is doing work on this lets work on it
together.
just got jprobe up and running, once I have ran a few tests, I will
experiment with the garbage collection, the memory is slowly growing, its
been running for an hour.
Filip
-----Original Message-----
From: Remy Maucherat [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 3:08 PM
To: Tomcat Developers List
Subject: Re: Memory leak- yeah I know
Filip Hanik wrote:
> Ok, I know that these emails are usually dismissed with the words:
> "Gets us some proof from a profiler", and I will...gimme some time, in
> the meanwhile, if anyone else wants to play with it.
>
> well the story is that during heavy load tests of the session
replication my
> system my VMs always run out of memory.
> What am I doing different than others, well, during my load tests I am
> not using keep alive connections to achieve true round robin. I'm
doing this by
> setting method.setHttp11(false) in the Http client.
>
> Running the test (attached, instructions below) I am hitting
> http://localhost:8080/index.jsp (home page) and watching the
memory grow and
> grow and grow.
>
> I am using a profiler and having somewhat of a hard time reading
the output.
> I am gonna download a trial of jprobe, see how that works out for
> me... I will be back with results from that.
>
> in the meantime, for those who want to see their memory grow here are
> the instructions
>
> My env:
> Machine: PIII 1.6Ghz, 512MB Ram
> OS: Redhat 8
> VM: Sun JDK 1.4.2_02 (build 1.4.2_02-b03)
>
> Instructions:
> download the attached client.jar
>
> execute:
> java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar
> org.apache.catalina.cluster.test.client.MemTestClient http
> ://192.168.0.103:7080/index.jsp 100 100000 1000 false (exchange ; for
> : on
> unix)
>
> Watching the memory grow using Top, although it shouldn't increase at
> all.
There's ab for that kind of stuff, you know ... No need to waste your time
writing some new stuff. Your tone is a bit sarcastic, I think. I'm eagerly
awaiting your "proofs" ;-)
Rémy
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
