Classloader not GC'ed after undeploying the applications. Reference from SingleElementLeafProperty --------------------------------------------------------------------------------------------------
Key: CXF-2689 URL: https://issues.apache.org/jira/browse/CXF-2689 Project: CXF Issue Type: Bug Components: JAX-WS Runtime Affects Versions: 2.2.6 Environment: Windows XP / Solaris 10 JBoss 5.1.0.GA Reporter: Danny Blazejczak I found that after undeploying my applications from JBoss 5.1.0.GA the classloaders remain hanging and are never garbage collected. These are the items I have tried so far: - JVM options : -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled These make that the permgen gets collected periodically. - I have verified that after several redeployments there is indeed the PermGen OOM error. The applications consist of a war package, containing the JSP frontend. and an EAR containing all the backend services (it includes CXF 2.2.6 and JAXB 2.1.12 within the EAR). I use CXF as a jaxws:client: Analyzing a memory dump several minutes after undeployment, and after manually triggering GC, I extracted this with Eclipse MAT. It contains the "Path to GC (Excluding all weak, soft and phantom references)": Class Name | Shallow Heap | Retained Heap --------------------------------------------------------------------------------------------------------------------------------------- org.jboss.classloader.spi.base.BaseClassLoader @ 0xdf9e538 | 96 | 10,484,208 '- <classloader> class com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty @ 0x2711fde0 | 8 | 8 |- <class> com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty @ 0xeeb0140 | 40 | 1,336 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x10173b88 | 32 | 32 | | '- [39] java.lang.ThreadLocal$ThreadLocalMap$Entry[128] @ 0xf7ca2f8 | 528 | 23,240 | | '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xf6647a8 | 24 | 23,264 | | '- threadLocals java.lang.Thread @ 0xf4856c0 http-127.0.0.1-8080-14 Native Stack, Thread| 104 | 23,928 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x101d8f98 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x101d9240 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x101e8df0 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x103c01f8 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x104c1b20 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x10567780 | 32 | 32 | '- Total: 7 entries | | |- <class> com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty @ 0xed70788 | 40 | 1,280 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xf8dcc78 | 32 | 32 | | '- [9] java.lang.ThreadLocal$ThreadLocalMap$Entry[128] @ 0xf7bca90 | 528 | 22,520 | | '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xf515ca8 | 24 | 22,544 | | '- threadLocals java.lang.Thread @ 0xf35e078 http-127.0.0.1-8080-8 Native Stack, Thread | 104 | 23,208 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xf8dcd18 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xf8e7628 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd624d0 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd6d480 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd77710 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd96740 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd9d5c0 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xff51780 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x10145a28 | 32 | 32 | |- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x106537c8 | 32 | 32 | '- Total: 11 entries | | '- Total: 2 entries | | --------------------------------------------------------------------------------------------------------------------------------------- It shows the SingleElementLeafProperty from JAXBImpl holding references to the classloader (of the EAR packaged application). the instance SingleElementLeafProperty contains a fieldName which holds one of the parameter names of the invoked operation. When I open a JPDA debug session to JBoss and suspend the Threads, I found indeed they included hard references to the SingleElementLeafProperty. This is a dump of the variables in the suspended Thread threadLocals: [25] ThreadLocal$ThreadLocalMap$Entry (id=229) discovered null next ThreadLocal$ThreadLocalMap$Entry (id=229) queue ReferenceQueue$Null (id=215) referent null value SingleElementLeafProperty<BeanT> (id=239) acc AdaptedAccessor<BeanT,InMemValueT,OnWireValueT> (id=240) defaultValue null fieldName "msisdn" (id=244) nillable false propertyInfo null tagName Name (id=245) xacc TransducedAccessor$CompositeTransducedAccessorImpl<BeanT,ValueT> (id=247) acc AdaptedAccessor<BeanT,InMemValueT,OnWireValueT> (id=240) adapter Class<T> (javax.xml.bind.annotation.adapters.NormalizedStringAdapter) (id=253) annotations null annotationType null cachedConstructor null classRedefinedCount 0 declaredAnnotations null declaredConstructors SoftReference<T> (id=349) declaredFields null declaredMethods null declaredPublicFields null declaredPublicMethods null enumConstantDirectory null enumConstants null genericInfo ClassRepository (id=279) lastRedefinedCount 0 name "javax.xml.bind.annotation.adapters.NormalizedStringAdapter" (id=283) newInstanceCallerCache null publicConstructors null publicFields null publicMethods null core Accessor$FieldReflection<BeanT,ValueT> (id=254) f Field (id=294) annotations byte[20] (id=298) clazz Class<T> (com.al.apc.generated.services.customeraccount.BaseAccountRequest) (id=300) declaredAnnotations LinkedHashMap<K,V> (id=301) fieldAccessor null genericInfo null modifiers 4 name "msisdn" (id=244) override true overrideFieldAccessor UnsafeObjectFieldAccessorImpl (id=303) root Field (id=305) securityCheckCache null securityCheckTargetClassCache null signature null slot 7 type Class<T> (java.lang.String) (id=209) valueType Class<T> (java.lang.String) (id=209) staticAdapter null valueType Class<T> (java.lang.String) (id=209) xducer RuntimeBuiltinLeafInfoImpl$1 (id=260) I have seen an old bug reference CXF-457 which clears the threadlocal at the end of invoke() method in JaxWsMethodInvoker. The proxy used in my case is JaxWsClientProxy. I found that after the method invocation the response is cleared from the threadlocals, but several parameter names remain there. I think my setup is OK. We passed a lot of tests, including 3M successful generated (test)load transactions. Any suggestions? Kind Regards, Danny -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.