Our application encountered class loading exceptions after upgrading to
Tomcat 10.1.33 (embedded), but only in the presence of the DataDog Java
Agent. Upgrading to Tomcat 10.1.34 seems to have resolved the issue
(thanks for the release!), so likely no further investigation is needed.
I'm reporting this just in case others in the community encounter the
problem or if the development team is curious to explore further.
We run with the DataDog Java Agent
(https://mvnrepository.com/artifact/com.datadoghq/dd-java-agent)
injected into our production deployments, command line example:
-javaagent:dd-java-agent-1.42.2.jar
This agent injects itself into the class loading process, altering
classes using Byte Buddy.
The first exception encountered while using 10.1.33 was when our
application attempted to load the GraalJS engine at startup, but we
narrowed down the problem and were able to reproduce it without GraalJS
in the mix, using a trivial static inner class that extends its outer class:
package org.labkey.devtools; // Use your favorite package
public class SystemThread extends Thread
{
public static final class InstrumentSystemThread extends
SystemThread
{
}
}
Elsewhere, attempt to load that inner class by name:
// This fails with a java.lang.LinkageError
Class.forName("org.labkey.devtools.SystemThread$InstrumentSystemThread");
This throws a java.lang.LinkageError: loader
org.labkey.embedded.LabKeySpringBootClassLoader @65557ca8 attempted
duplicate class definition for
org.labkey.devtools.SystemThread$InstrumentSystemThread.
(org.labkey.devtools.SystemThread$InstrumentSystemThread is in unnamed
module of loader org.labkey.embedded.LabKeySpringBootClassLoader
@65557ca8, parent loader
org.springframework.boot.devtools.restart.classloader.RestartClassLoader
@1410c78a)
Full stack trace of the DataDog code path is below. (Note that
LabKeySpringBootClassLoader is a simple subclass of Tomcat's
WebappClassLoader, used in this case to investigate the issue and log
stack traces at various steps. Other than selective logging, it doesn't
override any of the methods involved in class loading.)
Further investigation showed that explicitly loading the outer class
before the inner class resulted in a successful load:
// This succeeds
Class.forName("org.labkey.devtools.SystemThread");
Class.forName("org.labkey.devtools.SystemThread$InstrumentSystemThread");
Stepping through the class loading process was a bit tricky, given the
native code and Byte-Buddy-injected methods, but it was clear that the
inner class load began, a load of the outer class was attempted, and
then another (recursive) load of the inner class was attempted. I noted
that changes were made to Tomcat's WebappClassLoaderBase in 10.1.33 and
10.1.34, but I didn't pinpoint the specific changes that caused or fixed
this specific problem. Perhaps we're seeing another manifestation of
https://bz.apache.org/bugzilla/show_bug.cgi?id=69447, though the
symptoms are different and the workaround for that bug didn't work in
our case.
Let me know if you want further information. We're planning to happily
move on with Tomcat 10.1.34, since it seems to have addressed the issue.
Thanks,
Adam
Full stack trace logged by our loadClass() override, showing re-entrancy
of the GraalJS inner class:
at
org.labkey.bootstrap.LabKeyBootstrapClassLoader.loadClass(LabKeyBootstrapClassLoader.java:82)"
at
datadog.trace.agent.tooling.bytebuddy.outline.TypeFactory.loadType(TypeFactory.java:303)"
at
datadog.trace.agent.tooling.bytebuddy.outline.TypeFactory.lookupType(TypeFactory.java:276)"
at
datadog.trace.agent.tooling.bytebuddy.outline.TypeFactory.resolveType(TypeFactory.java:237)"
at
datadog.trace.agent.tooling.bytebuddy.outline.TypeFactory$LazyType.doResolve(TypeFactory.java:420)"
at
datadog.trace.agent.tooling.bytebuddy.outline.TypeFactory$LazyType.delegate(TypeFactory.java:414)"
at
net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.isLocalType(TypeDescription.java:8511)"
at
net.bytebuddy.description.type.TypeDescription$AbstractBase.isMemberType(TypeDescription.java:8283)"
at
net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining$WithFullProcessing$RedefinitionClassVisitor.onVisitInnerClass(TypeWriter.java:5255)"
at
net.bytebuddy.utility.visitor.MetadataAwareClassVisitor.visitInnerClass(MetadataAwareClassVisitor.java:257)"
at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:718)"
at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:425)"
at
net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:4014)"
at
net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2224)"
at
net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$UsingTypeWriter.make(DynamicType.java:4062)"
at
net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12529)"
at
net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12464)"
at
net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12173)"
at
net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12955)"
at
net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12885)"
at
net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)"
at
net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12407)"
at
datadog.trace.agent.tooling.bytebuddy.DDJava9ClassFileTransformer.transform(DDJava9ClassFileTransformer.java:60)"
at java.base/java.lang.ClassLoader.defineClass1(Native Method)"
at
org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:819)"
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1325)"
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1173)"
at
org.labkey.bootstrap.LabKeyBootstrapClassLoader.loadClass(LabKeyBootstrapClassLoader.java:84)"
at
datadog.trace.agent.tooling.bytebuddy.memoize.PreloadHierarchy.preload(PreloadHierarchy.java:46)"
at
datadog.trace.agent.tooling.bytebuddy.memoize.PreloadHierarchy.beforeDefineClass(PreloadHierarchy.java:28)"
at
datadog.trace.bootstrap.instrumentation.classloading.ClassDefining.begin(ClassDefining.java:14)"
at
org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:819)"
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1325)"
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1173)"
at
org.labkey.bootstrap.LabKeyBootstrapClassLoader.loadClass(LabKeyBootstrapClassLoader.java:84)"
at java.base/java.lang.Class.forName0(Native Method)"
at
com.oracle.truffle.api.impl.Accessor$Constants.loadSupport(Accessor.java:1379)"
at
com.oracle.truffle.api.impl.Accessor$Constants.<clinit>(Accessor.java:1368)"
at com.oracle.truffle.api.impl.Accessor.engineSupport(Accessor.java:1434)"
at
com.oracle.truffle.api.library.LibraryAccessor.engineAccessor(LibraryAccessor.java:57)"
at
com.oracle.truffle.api.library.LibraryFactory.loadExternalDefaultProviders(LibraryFactory.java:434)"
at
com.oracle.truffle.api.library.LibraryFactory.getExternalDefaultProviders(LibraryFactory.java:425)"
at
com.oracle.truffle.api.library.LibraryFactory.initDefaultExports(LibraryFactory.java:213)"
at
com.oracle.truffle.api.library.LibraryFactory.<init>(LibraryFactory.java:208)"
at
com.oracle.truffle.api.library.DynamicDispatchLibraryGen.<init>(DynamicDispatchLibraryGen.java:33)"
at
com.oracle.truffle.api.library.DynamicDispatchLibraryGen.<clinit>(DynamicDispatchLibraryGen.java:25)"
at java.base/java.lang.Class.forName0(Native Method)"
at
com.oracle.truffle.api.library.LibraryFactory.loadGeneratedClass(LibraryFactory.java:770)"
at
com.oracle.truffle.api.library.LibraryFactory.resolveImpl(LibraryFactory.java:751)"
at
com.oracle.truffle.api.library.LibraryFactory.resolve(LibraryFactory.java:744)"
at
com.oracle.truffle.api.library.LibraryFactory.<init>(LibraryFactory.java:202)"
at
com.oracle.truffle.api.interop.InteropLibraryGen.<init>(InteropLibraryGen.java:180)"
at
com.oracle.truffle.api.interop.InteropLibraryGen.<clinit>(InteropLibraryGen.java:171)"
at java.base/java.lang.Class.forName0(Native Method)"
at
com.oracle.truffle.api.library.LibraryFactory.loadGeneratedClass(LibraryFactory.java:770)"
at
com.oracle.truffle.api.library.LibraryFactory.resolveImpl(LibraryFactory.java:751)"
at
com.oracle.truffle.api.library.LibraryFactory.resolve(LibraryFactory.java:744)"
at
com.oracle.truffle.api.interop.InteropLibrary.<clinit>(InteropLibrary.java:2972)"
at
com.oracle.truffle.polyglot.PolyglotValueDispatch.<clinit>(PolyglotValueDispatch.java:168)"
at
com.oracle.truffle.polyglot.PolyglotImpl.initialize(PolyglotImpl.java:202)"
at org.graalvm.polyglot.Engine.loadAndValidateProviders(Engine.java:1691)"
at org.graalvm.polyglot.Engine$1.run(Engine.java:1717)"
at org.graalvm.polyglot.Engine$1.run(Engine.java:1712)"
at org.graalvm.polyglot.Engine.initEngineImpl(Engine.java:1712)"
at org.graalvm.polyglot.Engine$ImplHolder.<clinit>(Engine.java:170)"
at org.graalvm.polyglot.Engine.getImpl(Engine.java:422)"
at
org.graalvm.polyglot.HostAccess$Builder.targetTypeMapping(HostAccess.java:1283)"
at
com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.createNashornHostAccess(GraalJSScriptEngine.java:107)"
at
com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.<clinit>(GraalJSScriptEngine.java:102)"
at
com.oracle.truffle.js.scriptengine.GraalJSEngineFactory.getScriptEngine(GraalJSEngineFactory.java:151)"
at
org.labkey.core.reports.ScriptEngineManagerImpl.getEngineByName(ScriptEngineManagerImpl.java:212)"
at
org.labkey.core.wiki.MarkdownServiceImpl$PoolFactory.makeObject(MarkdownServiceImpl.java:73)"
at
org.labkey.core.wiki.MarkdownServiceImpl$PoolFactory.makeObject(MarkdownServiceImpl.java:51)"
at
org.apache.commons.pool.impl.StackKeyedObjectPool.borrowObject(StackKeyedObjectPool.java:165)"
at
org.labkey.core.wiki.MarkdownServiceImpl.toHtml(MarkdownServiceImpl.java:168)"
at
org.labkey.core.wiki.MarkdownServiceImpl.toHtml(MarkdownServiceImpl.java:147)"
at
org.labkey.core.wiki.MarkdownServiceImpl.<init>(MarkdownServiceImpl.java:132)"
at
org.labkey.core.CoreModule.startupAfterSpringConfig(CoreModule.java:1212)"
at org.labkey.api.module.SpringModule.doStartup(SpringModule.java:120)"
at org.labkey.api.module.DefaultModule.startup(DefaultModule.java:231)"
at
org.labkey.api.module.ModuleLoader.completeStartup(ModuleLoader.java:1664)"
at
org.labkey.api.module.ModuleLoader.initiateModuleStartup(ModuleLoader.java:1603)"
at org.labkey.api.module.ModuleLoader.afterUpgrade(ModuleLoader.java:1840)"
at
org.labkey.api.module.ModuleLoader.lambda$startNonCoreUpgradeAndStartup$17(ModuleLoader.java:1828)"
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
