Hi John I had a similar situation with Azure Application Insights. Had it as a maven dependency built into the uber JAR, but ran into NoClassDef found. I ended up packaging it to a custom Docker image, and that works as expected.
Oliver > > Am 23.11.2021 um 20:18 schrieb John Karp <johnk...@yahooinc.com>: > > Hi, > > Due to limitations with Flink's metrics API* I'm using Micrometer to > collect supplemental application metrics in my Flink 1.13 application > (hosted on Kinesis Data Analytics), and publish them to AWS > CloudWatch. For the most part it works well. However when handling > errors from CloudWatch, the worker thread hits > java.lang.NoClassDefFoundError. > > This is despite the classes in question being included in the > application JAR (which I confirmed by dumping the JAR contents.) I > hold the Micrometer registry in a "private static final" field, which > may be part of the issue; I want the metric registry to be a singleton > in the JVM to minimize costs incurred publishing to CloudWatch. > (Micrometer is fully thread-safe.) > > I know this metrics approach isn't very standard in the Flink world, > but perhaps is there a way it can be made to work? (Note: this is > hosted on Kinesis Data Analytics where the Flink configuration cannot > generally be changed.) > > Exception 1: > > java.lang.NoClassDefFoundError: > software/amazon/awssdk/core/internal/util/ThrowableUtils > at > software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$doExecute$4(BaseAsyncClientHandler.java:222) > ~[?:?] > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) > ~[?:?] > ... 42 more > Wrapped by: java.util.concurrent.CompletionException: > java.lang.NoClassDefFoundError: > software/amazon/awssdk/core/internal/util/ThrowableUtils > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) > ~[?:?] > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) > ~[?:?] > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:932) > ~[?:?] > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) > ~[?:?] > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) > ~[?:?] > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) > ~[?:?] > at > software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:54) > ~[?:?] > ... > at > software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:209) > ~[?:?] > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) > ~[?:?] > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) > ~[?:?] > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > ~[?:?] > at java.lang.Thread.run(Thread.java:829) ~[?:?] > > Exception 2: > > java.lang.IllegalStateException: Trying to access closed classloader. > Please check if you store classloaders directly or indirectly in > static fields. If the stacktrace suggests that the leak occurs in a > third party library and cannot be fixed immediately, you can disable > this check with the configuration > 'classloader.check-leaked-classloader'. > at > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:169) > ~[flink-dist_2.12-1.13.2.jar:1.13.2] > at > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResources(FlinkUserCodeClassLoaders.java:193) > ~[flink-dist_2.12-1.13.2.jar:1.13.2] > at > java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1196) > ~[?:?] > at > java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221) > ~[?:?] > at > java.util.ServiceLoader$LazyClassPathLookupIterator$1.run(ServiceLoader.java:1268) > ~[?:?] > at > java.util.ServiceLoader$LazyClassPathLookupIterator$1.run(ServiceLoader.java:1267) > ~[?:?] > at java.security.AccessController.doPrivileged(Native Method) ~[?:?] > at > java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1270) > ~[?:?] > at java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300) ~[?:?] > at java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385) ~[?:?] > at javax.xml.stream.FactoryFinder$1.run(FactoryFinder.java:348) ~[?:?] > at java.security.AccessController.doPrivileged(Native Method) ~[?:?] > at > javax.xml.stream.FactoryFinder.findServiceProvider(FactoryFinder.java:337) > ~[?:?] > at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:309) ~[?:?] > at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:222) ~[?:?] > at > javax.xml.stream.XMLInputFactory.newInstance(XMLInputFactory.java:161) > ~[?:?] > at > software.amazon.awssdk.protocols.query.unmarshall.XmlDomParser.createXmlInputFactory(XmlDomParser.java:124) > ~[?:?] > at > java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305) > ~[?:?] > at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195) ~[?:?] > at java.lang.ThreadLocal.get(ThreadLocal.java:172) ~[?:?] > at > software.amazon.awssdk.protocols.query.unmarshall.XmlDomParser.parse(XmlDomParser.java:44) > ~[?:?] > at java.util.Optional.map(Optional.java:265) ~[?:?] > at > software.amazon.awssdk.protocols.query.internal.unmarshall.QueryProtocolUnmarshaller.unmarshall(QueryProtocolUnmarshaller.java:72) > ~[?:?] > ... > at > software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495) > ~[?:?] > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > ~[?:?] > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) > ~[?:?] > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) ~[?:?] > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > ~[?:?] > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > ~[?:?] > ... 1 more > Wrapped by: software.amazon.awssdk.core.exception.SdkClientException: > Unable to execute HTTP request: Trying to access closed classloader. > Please check if you store classloaders directly or indirectly in > static fields. If the stacktrace suggests that the leak occurs in a > third party library and cannot be fixed immediately, you can disable > this check with the configuration > 'classloader.check-leaked-classloader'. > at > software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) > ~[?:?] > at > software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) > ~[?:?] > at > software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:198) > ~[?:?] > at > software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:194) > ~[?:?] > at > software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:143) > ~[?:?] > ... 22 more > Wrapped by: java.util.concurrent.CompletionException: > software.amazon.awssdk.core.exception.SdkClientException: Unable to > execute HTTP request: Trying to access closed classloader. Please > check if you store classloaders directly or indirectly in static > fields. If the stacktrace suggests that the leak occurs in a third > party library and cannot be fixed immediately, you can disable this > check with the configuration 'classloader.check-leaked-classloader'. > at > software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:60) > ~[?:?] > at > software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51) > ~[?:?] > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) > ~[?:?] > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) > ~[?:?] > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) > ~[?:?] > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) > ~[?:?] > at > software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:74) > ~[?:?] > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) > ~[?:?] > ... > at > software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:209) > ~[?:?] > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) > ~[?:?] > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) > ~[?:?] > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > ~[?:?] > at java.lang.Thread.run(Thread.java:829) ~[?:?] > > > Thanks, > John > > * > http://mail-archives.apache.org/mod_mbox/flink-user/202108.mbox/%3c06981cbb-d71b-dc3b-7dba-6a2eeab2c...@apache.org%3e
