Hi, Have you checked build of your job for a dependency convergence errors? Either automatically or manually (`mvn dependency:tree` command)? Look for the version clashes for the dependencies that are pulling in `io/grpc/netty/shaded/io/netty/channel/AbstractChannel` class (grpc-netty?).
It could be an error purely with dependencies of your job. Alternatively if you are using `flink-connector-gcp-pubsub` connector, it can be a clash between something that you are pulling in and this connector (I think this is the only thing from Flink that depends on grpc-netty). Either way, if it’s a dependency conflict, you have to fix it somehow. As long as your application depends on different versions of the same dependency you might always keep getting class/method not found errors. Potential solutions: - drop the dependency - converge the versions of this dependency (enforce your build system to always use the same version of it in all of the places) - shade it Piotrek > On 22 Oct 2019, at 10:40, dhanesh arole <davcdhane...@gmail.com> wrote: > > Just to give you more context, > > We are using `com.google.cloud.bigtable` as well in our job dep. Could it be > due to shaded plugin issue with `bigtable-hbase-2.x` ? > > > - > Dhanesh Arole ( Sent from mobile device. Pardon me for typos ) > > > > On Tue, Oct 22, 2019 at 2:06 PM dhanesh arole <davcdhane...@gmail.com > <mailto:davcdhane...@gmail.com>> wrote: > Hello all, > > We are running on Flink 1.6.2 and have a couple of streaming jobs running on > it. We are intermittently seeing Java.lang.NoClassDefFoundError with below > stack trace[1]. The Flink job was working fine but due to recent this errors, > task managers are now frequently crashing and causing to restart the job. We > haven't changed any dep version / Flink version, so we are not sure if it's > related to version mismatch. This is only happening with classes from > io.grpc.netty.shaded package. Even the classes that FlinkClassLoader is not > able to find are actually present in fat jar built using shadowJar - > (validated by looking at jar tf flink-job.jar). > > Has anyone faced such issue before? Any pointers on how to debug this would > be really helpful. > > I am attaching output of './gradlew dep' as well for reference, so as to > validate if we don't have some wrong or flaky dependencies. > > [1] Stack trace: Task manager Java.lang.NoClassDefFoundError > > "java.lang.NoClassDefFoundError: > io/grpc/netty/shaded/io/netty/channel/AbstractChannel$AbstractUnsafe$8 > at > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.deregister(AbstractChannel.java:817) > at > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.fireChannelInactiveAndDeregister(AbstractChannel.java:777) > at > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:760) > at > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615) > at > io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105) > at > io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171) > at > io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) > at > io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591) > at > io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) > at > io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) > at > io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) > at > io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$8 > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 13 more > "
