Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-18 Thread Robert Metzger
Yeah, that's the problem Your Kafka 0.8.1 dependency [1] doesn't depend on kafka-clients. With the explicit dependency definition, you are overriding the transitive Kafka dependency from the kafka connector. [1] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.1/kafka_2.10-0.8.1.pom

Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-18 Thread Robert Schmidtke
Turns out when I remove the explicit dependency on kafka_2.10 v. 0.8.1, then the dependencies are properly included. Guess there was a conflict somehow? I'll need to figure out if the rest of the code is fine with kafka_2.10 v. 0.8.2.0 as well. On Mon, Apr 18, 2016 at 4:32 PM, Robert Schmidtke wr

Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-18 Thread Robert Schmidtke
Hi Robert, thanks for your offer. After playing around a bit I would like to take it, if you have the time: https://github.com/robert-schmidtke/HiBench/blob/flink-streaming/src/streambench/flinkbench/pom.xml I would guess the POM is similar to the one in the sample project, yet when building it,

Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-18 Thread Robert Metzger
If you want you can share the pom of your project privately. On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke wrote: > You're right, it does not. When including it the resulting jar has the > Kafka dependencies bundled. Now it's up to me to figure out the difference > between the sample project

Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-18 Thread Robert Schmidtke
You're right, it does not. When including it the resulting jar has the Kafka dependencies bundled. Now it's up to me to figure out the difference between the sample project and the one I'm working on. Thanks! Really quick help. Robert On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger wrote: > Hi

Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-18 Thread Robert Metzger
Hi, the problem with the posted project is that it doesn't have the Flink kafka connector as a dependency. On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke wrote: > Hi Robert, > > thanks for your hints. I was not sure whether I was building a proper fat > jar, as I have not used the Flink Arche

Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-18 Thread Robert Schmidtke
Hi Robert, thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at https://github.com/robert-schmidtke/flink-test/ which is nothing more than the Quickstart Archetype plus the

Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-18 Thread Robert Metzger
Hi, did you check your user jar if it contains the Kafka classes? Are you building a fat jar? Are you manually excluding any dependencies? Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contain

Re: Flink 0.10.2 and Kafka 0.8.1

2016-04-17 Thread Balaji Rajagopalan
I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2 scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. balaji On Mon, Apr 18,

Flink 0.10.2 and Kafka 0.8.1

2016-04-17 Thread Robert Schmidtke
Hi everyone, I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/