Thanks for your response Chris. On Fri, Oct 8, 2010 at 10:07 PM, Christopher Schultz < ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Anurag, > > On 10/8/2010 3:15 PM, Anurag Kapur wrote: > > _Problem Statement_ > > Memory leak in web application running on Tomcat > > _Solution Statement_ > > Fix your memory leak > That's the plan :) > > _System Information_ > > > > Tomcat 5.5.27 > > This statement plus the subject line are confusing me. > Apologies. I have made a typo in the subject line. I am using Tomcat 5.5.27 > _Observations and Investigation Done_ > > > > Under load, the web application slowly runs out of heap space. The heap > > utilization graph suggests a memory leak type pattern. > > Attachments are stripped: there is no image to view, here. I have attached the heap usage graph as a file this time (heap_usage.jpg) > Heap dumps were gathered to determine the root cause and observations > > are as below from the dumps: > > > > 1. HashMap entries from the below object reference tree seem to > > consume over 80% of the used tenured generation space. > > > > org/apache/jasper/compiler/JspRuntimeContext > > Java/util/Collections$SynchronizedMap > > Java/util/HashMap > > Java/util/HashMap$Entry > > If this doesn't grow, then it doesn't sound like a memory leak. > The number of entries under the HashMap referenced by JspRuntimeContent -> Collections$SynchronizedMap does not increase. However the size of these 100 entries increases with time. In my last test, after the first full GC the size of this HashMap was around 350MB. With time and under load, this increased. Next heap dump gathered after the 4th Full GC showed that the size of this hash map had doubled to around 730MB. The number of entries immediately under this HashMap however remained constant. > > > 2. Heap dumps gathered during different times and after a Full GC > > always indicate 100 entries > > This indicates zero growth, so I would guess no memory leak. > Yes, the number of immediate entries in the Hashmap remained constant however the size these occupy in the heap increased. > > > 3. The number of objects referenced by each HashMap entry vary > > between 2-3 > > 1. These are either a String and JspServletWrapper object > > > > or > > > > 2. A String, JspServletWrapper and _another HashMap Entry_. > > This call reference tree of HashMap entries referenced by > > another HashMap entry can repeat to a depth of approximately > > 8-10 nodes > > That sounds weird. > To make this a little clear, I have attached a screen shot of the object reference tree from IBM Heap analyzer I used on the heap dumps obtained. (object_reference_tree.jpg) > > > The above is indicated in an object reference tree obtained after > > analyzing the Heap dump as shown below > > > > 4. The maximum percentage of the memory occupied by the HashMap entry > > object is by a character array that seems like the JSP servlet > > response. HTML response (tags with content) can be seen in the > > character array > > Does this response text stay in memory indefinitely? When you say it's > the response, is it the actual response to a particular client, or is it > just static text from the JSP source? If the latter, this is completely > expected: the JSP compiles-in all static text I am having difficulties at this stage determining this with 100% confidence. I am mostly seeing static HTML tags in the character array that would be coming from the static JSP source. However I have also seen some references to dynamic content that is sent back to the client in these charcter arrays. There is atleast 1 such character array in each of these 100 has map entries. Sometime more in cases where there are more hash map entries within an entry. I have not seen the size of all these charcter arrays but a few are as big as 7MB. . > > > _Questions_ > > > > 1. I am stuck at this point and have run out of ideas on how to get > > to the root cause of this issue. Do you have any ideas/suggestions > > to help identify the root cause? > > Finding the roots to those references should help. For instance, what > page is holding on to all those entries? It's possible that the > JSPRuntimeContext requires these data even after the compiler has > compiled the code. > I can also see the name of the custom JSP files in the object call reference trees under most of these 100 hash map entries. However some of them are relatively simple and I am unable to think of a reason why such a JSP would hold reference in the heap causing the leak. As you suggested, I am now trying to search for objects I see in the heap dump obtained after the first Full GC in the dump obtained after the 4th Full GC. Thanks for your inputs so far. Any other inputs while I do the investigation highlighted above? > 2. I google and found the following interesting links > > 1. > http://www.mail-archive.com/d...@tomcat.apache.org/msg03395.html > > Is there any know issues, configuration that causes a memory > > leak by caching of servlet responses in the container and > > not flushing the cached objects? > > The above seems to be a complaint about how JSPs work: they are > translated into .java files, then compiled and loaded. These loaded > class files contain lots of static strings. There's nothing to be done > about this. > > > 2. > http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3c000c01c2ee2c$41c079a0$03b69...@garethdqw0t9if%3e > > We use a lot of JSTL in our web application. Are there any > > known issues around memory issues as indicated on the link? > > This one is seriously old: 2003 was a long time ago (even older than > your version of Tomcat). I don't think that post is relevant anymore. > > You should consider upgrading. Check the changelog > (http://tomcat.apache.org/tomcat-5.5-doc/changelog.html) for changes to > Jasper between your version (5.5.25 or 5.5.27) and the latest (5.5.30). > You also might consider upgrading to Tomcat 6.0. > > - -chris > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (MingW32) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkyviC8ACgkQ9CaO5/Lv0PCHQQCeKH/X4Ee4nN1Z3x19wCNkWW3Q > pf0An2e+6trJtZezARIhx3GXifLUVpWc > =Id6K > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
