This is long and mostly off-topic. Read if you wish to follow the fun
and games of tracking down permgen leaks, especially with a third party
obfuscated library.
On 8/27/2012 8:15 AM, Christopher Schultz wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 8/25/12 2:15 PM, Mark Eggers wrote:
I could even work on convincing people to move away from this PDF
library (creates permgen leaks when images are included) to
something a bit more sane. Maybe even something that read stream
data and didn't cause a permgen memory leak . . . .
There have been some recent updates to the
JreMemoryLeakPreventionListener that may help alleviate these. Search
for "AWT" in the JMLPL documentation.
Also, if you've found something that you can trace back to the JVM,
please let us know and we can update the JMLPL to help avoid PermGen
leaks.
- -chris
Chris,
JMLPL documentation?
I'm also stuck with 6.0.x (where x is 35 in development, and 20 in
production). I do not think that upgrading production is a possibility.
However, I'm seeing this in 6.0.35.
A custom context listener has been used to pin the AWT thread to the
system class loader. This was based on comments made in the Tomcat
mailing list a while back. The code is as follows:
public class GraphicsListener implements ServletContextListener {
private Log log =
LogFactory.getLog(org.mdeggers.listeners.GraphicsListener.class);
public void contextInitialized(ServletContextEvent sce) {
if (log.isInfoEnabled()) {
log.info("Pinning AWT Event Thread to parent classloader.");
}
// get the current classloader for the thread and store it
Thread myThread = Thread.currentThread();
ClassLoader ccl = myThread.getContextClassLoader();
// switch to the system classloader and start up the AWT thread
myThread.setContextClassLoader(ClassLoader.getSystemClassLoader());
EventQueue eq = Toolkit.getDefaultToolkit().getSystemEventQueue();
// switch back to the current classloader
myThread.setContextClassLoader(ccl);
}
public void contextDestroyed(ServletContextEvent sce) {
}
}
This was done since the target server was not known (at the time). Also,
this doesn't require changes to a Tomcat server that is not in my control.
Using the following leak prevention listener configuration didn't change
the fact that there was a leak on unloading the application.
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
AWTThreadProtection="true"/>
Note that by default I have this configured in my environment.
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
According to the documentation, this takes care of many leak issues,
including AppContext.
Now the fun begins. I first run this under Tomcat 6.0.35 and JRE
1.6.0_32 on Windows 7 Home Premium (64 bit).
Tracking down one of the leaked library classes, I get this.
--> class sun.awt.AppContext (160 bytes) (static field mainAppContext:)
--> sun.awt.AppContext@0xa0126130 (81 bytes) (field table:)
--> java.util.HashMap@0xa0126168 (64 bytes) (field table:)
--> [Ljava.util.HashMap$Entry;@0xa0126198 (144 bytes)
(Element 1 of [Ljava.util.HashMap$Entry;@0xa0126198:)
--> java.util.HashMap$Entry@0xa04cc760 (44 bytes) (field value:)
--> java.awt.MediaTracker@0xa04cc790 (32 bytes) (field target:)
--> javax.swing.ImageIcon$3@0xa04cc7a8 (379 bytes) (field acc:)
--> java.security.AccessControlContext@0xa04cc888 (41 bytes)
(field context:)
--> [Ljava.security.ProtectionDomain;@0xa04cc8a8 (80 bytes)
(Element 3 of [Ljava.security.ProtectionDomain;@0xa04cc8a8:)
--> java.security.ProtectionDomain@0xa0424dc8 (58 bytes)
(field classloader:)
--> org.apache.catalina.loader.WebappClassLoader@0xa04205d0 (309 bytes)
Now, I run this under Tomcat 7.0.27 and JRE 1.7.0._05 on Windows 7 Home
Premium (64 bit).
Tracking down one of the leaked library classes, I get this.
--> class sun.awt.AppContext (168 bytes) (static field mainAppContext:)
--> sun.awt.AppContext@0xa0255698 (81 bytes) (field table:)
--> java.util.HashMap@0xa02556d0 (64 bytes) (field table:)
--> [Ljava.util.HashMap$Entry;@0xa0255700 (144 bytes)
(Element 6 of [Ljava.util.HashMap$Entry;@0xa0255700:)
--> java.util.HashMap$Entry@0xa0d81ca8 (44 bytes) (field value:)
--> java.awt.MediaTracker@0xa0d81d28 (32 bytes) (field target:)
--> javax.swing.ImageIcon$3@0xa0d81d40 (364 bytes) (field acc:)
--> java.security.AccessControlContext@0xa0d81e18 (41 bytes)
(field context:)
--> [Ljava.security.ProtectionDomain;@0xa0d81e38 (80 bytes)
(Element 3 of [Ljava.security.ProtectionDomain;@0xa0d81e38:)
--> java.security.ProtectionDomain@0xa0c14c58 (58 bytes)
(field classloader:)
--> org.apache.catalina.loader.WebappClassLoader@0xa0b135c8 (325 bytes)
Looks pretty much the same.
These are all chains present in the library (not the test application
that I wrote).
Here's how I arrived at the call chain.
0. Run the application and hit a PDF page that includes an image
1. Unload the application
2. Force a garbage collection with VisualVM
3. Do a heap dump with VisualVM
4. Save the heap dump
5. Use jhat to look at the heap dump
a. Find a leaked class
b. Click on its classloader
c. Reference chains from rootset - exclude weak references
d. scroll down to system class references
However, there are now clues in catalina.out (Tomcat 7).
Relative URL
------------
SEVERE: Servlet.service() for servlet default threw exception
java.io.FileNotFoundException:
The requested resource
(/PTest/pjsp/C:/Users/mdeggers/Apache/apache-tomcat-7.0.27/webapps/PTest/images/gf-title-sm.jpg)
is not available
Absolute (file:///) URL
-----------------------
SEVERE: Servlet.service() for servlet default threw exception
java.io.FileNotFoundException:
The requested resource
(/PTest/c:/Users/mdeggers/Apache/apache-tomcat-7.0.27/temp/gf-title-sm.jpg)
is not available
I apologize for the line wrapping.
In both cases, the physical location of the image file is shown
correctly. However, in the first case (relative URL) the context (PTest)
and the path to the JSP (pjsp) are added to the front. In the second
case (file:/// url), only the context (PTest) is added.
I've not posted the entire stack trace so I don't incriminate the makers
of the third party library. The stack trace is indeed different between
the relative and absolute cases.
I will post the entire stack trace on their forum and help mailing list.
I think the library does some brute force work in order to load images.
The first is the file path shown above. The second appears to be a
getResourceAsStream. Since there are icons in the JAR files supplied by
the library, these appear to be loaded. The next method (whatever that
is), appears to load the images.
The impact is slight with the test application. With the production
application there is about 20 MB of memory consumed per application reload.
. . . . what a mess.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
