Re: Loading FlinkKafkaProducer fails with LinkError

Hi, @Chesnay Schepler The issue is that the uber-jar is first loaded with Flink's app classloader (because it's in lib) and then when the application starts, it gets loaded again in the ChildFirstCL and since it's child-first, the class is loaded anyways. What I don't quite understand is why the

Re: Loading FlinkKafkaProducer fails with LinkError

They are definitely equal, the same JAR is copied in subsequent lines in the Dockerfile. Regarding the NoSuchMethodException, I'll look it up and let you know tomorrow. On Tue, Aug 25, 2020, 22:59 Chesnay Schepler wrote: > The simplest answer is that they are in fact not equal; maybe it is a ja

Re: Loading FlinkKafkaProducer fails with LinkError

The simplest answer is that they are in fact not equal; maybe it is a jar of an older version of your setup? Can you give some details on the NoSuchMethodException? Specifically whether it tries to access something from the Kafka connector, or from your own user code. On 25/08/2020 21:27, Yu

Re: Loading FlinkKafkaProducer fails with LinkError

OK, I think I figured it out. It looks like the uber-jar is also being placed under `lib`, which is probably the cause of the problem. Question is, why does it identify it as two different versions? It's exactly the same JAR. On Tue, Aug 25, 2020 at 10:22 PM Yuval Itzchakov wrote: > I'm afraid

Re: Loading FlinkKafkaProducer fails with LinkError

I'm afraid it's not being printed out due to different log levels :( Yes, I build the image myself. It takes the tar file from https://archive.apache.org/dist/flink/flink-1.9.0/ and unpacks it into the image. I've ran: find . -iname "*.jar" | x

Re: Loading FlinkKafkaProducer fails with LinkError

The NoSuchMethodException shows that the class is still on the classpath, but with a different version than your code is expecting. Otherwise you would've gotten a different error. This implies that there are 2 versions of the kafka dependencies on the classpath in your original run; it suddenly

Re: Loading FlinkKafkaProducer fails with LinkError

Will it be enough to provide you the output of `-verbose:class`? Or do you want me to add additional arguments? On Tue, Aug 25, 2020 at 6:20 PM Arvid Heise wrote: > Small correction: you'd bundle the connectors in your uber jar like you > did but you usually don't put it into flink-dist. > > So

Re: Loading FlinkKafkaProducer fails with LinkError

Hi Arvid, I'm running Flink in a job cluster on k8s using the Lyft Operator. The flink image that I'm building does not have the flink-connector-kafka library in it's JAR, I've made sure of this using `jar -tf`. Additionally, once I removed the dependency from my uber jar, it failed with a "NoSuch

Re: Loading FlinkKafkaProducer fails with LinkError

Small correction: you'd bundle the connectors in your uber jar like you did but you usually don't put it into flink-dist. So please double-check if it's also in flink-dist and remove it there. If not, then please add the full classpath log statement. It might also be a bug related to restoring an

Re: Loading FlinkKafkaProducer fails with LinkError

Hi Yuval, How do you execute Flink? Can you show us the log entry with the classpath? I'm guessing that you have Kafka bundled in your uber-jar and additionally also have the connector in flink-dist/lib. If so, you simply need to remove it in one place. In general, if you use flink-dist, you'd no

Loading FlinkKafkaProducer fails with LinkError

Hi, I'm trying to load a FlinkKafkaProducer sink alongside another custom sink. While trying to restore a running Flink app from the previous state, I get the error message below. I am running Flink 1.9.0 with the following SBT dependency added: "org.apache.flink" %% "flink-connector-kafka" % 1.9