Just to give you more context, We are using `com.google.cloud.bigtable` as well in our job dep. Could it be due to shaded plugin issue with `bigtable-hbase-2.x` ?
- Dhanesh Arole ( Sent from mobile device. Pardon me for typos ) On Tue, Oct 22, 2019 at 2:06 PM dhanesh arole <davcdhane...@gmail.com> wrote: > Hello all, > > We are running on Flink 1.6.2 and have a couple of streaming jobs running > on it. We are intermittently seeing *Java.lang.NoClassDefFoundError* with > below stack trace[1]. The Flink job was working fine but due to recent this > errors, task managers are now frequently crashing and causing to restart > the job. We haven't changed any dep version / Flink version, so we are not > sure if it's related to version mismatch. This is only happening with > classes from *io.grpc.netty.shaded* package. Even the classes that > FlinkClassLoader is not able to find are actually present in fat jar built > using shadowJar - (validated by looking at *jar tf flink-job.jar).* > > Has anyone faced such issue before? Any pointers on how to debug this > would be really helpful. > > I am attaching output of './gradlew dep' as well for reference, so as to > validate if we don't have some wrong or flaky dependencies. > > [1] Stack trace: Task manager Java.lang.NoClassDefFoundError > > "java.lang.NoClassDefFoundError: > io/grpc/netty/shaded/io/netty/channel/AbstractChannel$AbstractUnsafe$8 > at > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.deregister(AbstractChannel.java:817) > at > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.fireChannelInactiveAndDeregister(AbstractChannel.java:777) > at > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:760) > at > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615) > at > io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105) > at > io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171) > at > io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) > at > io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591) > at > io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) > at > io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) > at > io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) > at > io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$8 > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 13 more > " >
