Hi Arvid, I'm running Flink in a job cluster on k8s using the Lyft Operator.
The flink image that I'm building does not have the flink-connector-kafka library in it's JAR, I've made sure of this using `jar -tf`. Additionally, once I removed the dependency from my uber jar, it failed with a "NoSuchMethodException" at runtime for one of the arbitrary methods. I used classloader.resolve-order: parent-first and it resolved the issue somehow. I still don't know why though. On Tue, Aug 25, 2020 at 6:13 PM Arvid Heise <ar...@ververica.com> wrote: > Hi Yuval, > > How do you execute Flink? Can you show us the log entry with the classpath? > > I'm guessing that you have Kafka bundled in your uber-jar and additionally > also have the connector in flink-dist/lib. If so, you simply need to remove > it in one place. In general, if you use flink-dist, you'd not bundle any > Flink dependencies in your uber-jar (use provided scope for them). > > If you have everything bundled in one uber-jar and execute it somehow > without flink-dist, then I don't immediately see a solution. Then the log > with the classpath would help. > > Best, > > Arvid > > > On Sun, Aug 23, 2020 at 1:37 PM Yuval Itzchakov <yuva...@gmail.com> wrote: > >> Hi, >> I'm trying to load a FlinkKafkaProducer sink alongside another custom >> sink. While trying to restore >> a running Flink app from the previous state, I get the error message >> below. >> >> I am running Flink 1.9.0 with the following SBT dependency added: >> >> "org.apache.flink" %% "flink-connector-kafka" % 1.9.0 >> >> And the app is deployed via a standard uber jar with all the >> dependencies. W >> Would appreciate the help >> >> java.lang.LinkageError: loader constraint violation: loader (instance of >> org/apache/flink/util/ChildFirstClassLoader) previously initiated >> loading for a different type with name >> "org/apache/kafka/clients/producer/ProducerRecord" >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:763) >> at java.security.SecureClassLoader.defineClass(SecureClassLoader >> .java:142) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) >> at java.net.URLClassLoader.access$100(URLClassLoader.java:74) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:369) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:363) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:362) >> at org.apache.flink.util.ChildFirstClassLoader.loadClass( >> ChildFirstClassLoader.java:66) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at java.lang.Class.getDeclaredMethods0(Native Method) >> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) >> at java.lang.Class.getDeclaredMethod(Class.java:2128) >> at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java: >> 1629) >> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:79) >> at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:520) >> at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494) >> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391) >> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681) >> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java: >> 1885) >> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java: >> 1751) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream >> .java:2042) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream >> .java:2287) >> at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream >> .java:561) >> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer >> .readObject(FlinkKafkaProducer.java:1202) >> at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke( >> DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java: >> 1170) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java: >> 2178) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream >> .java:2069) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream >> .java:2287) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java: >> 2211) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream >> .java:2069) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream >> .java:2287) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java: >> 2211) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream >> .java:2069) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) >> at org.apache.flink.util.InstantiationUtil.deserializeObject( >> InstantiationUtil.java:576) >> at org.apache.flink.util.InstantiationUtil.deserializeObject( >> InstantiationUtil.java:562) >> at org.apache.flink.util.InstantiationUtil.deserializeObject( >> InstantiationUtil.java:550) >> at org.apache.flink.util.InstantiationUtil.readObjectFromConfig( >> InstantiationUtil.java:511) >> at org.apache.flink.streaming.api.graph.StreamConfig >> .getStreamOperatorFactory(StreamConfig.java:235) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createChainedOperator(OperatorChain.java:427) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createOutputCollector(OperatorChain.java:354) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createChainedOperator(OperatorChain.java:418) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createOutputCollector(OperatorChain.java:354) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createChainedOperator(OperatorChain.java:418) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createOutputCollector(OperatorChain.java:354) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createChainedOperator(OperatorChain.java:418) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createOutputCollector(OperatorChain.java:354) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createChainedOperator(OperatorChain.java:418) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createOutputCollector(OperatorChain.java:354) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createChainedOperator(OperatorChain.java:418) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createOutputCollector(OperatorChain.java:354) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createChainedOperator(OperatorChain.java:418) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createOutputCollector(OperatorChain.java:354) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createChainedOperator(OperatorChain.java:418) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain >> .createOutputCollector(OperatorChain.java:354) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>( >> OperatorChain.java:144) >> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke( >> StreamTask.java:370) >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) >> at java.lang.Thread.run(Thread.java:748) >> >> >> -- >> Best Regards, >> Yuval Itzchakov. >> > > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng > -- Best Regards, Yuval Itzchakov.
