Yeah, that's the problem Your Kafka 0.8.1 dependency [1] doesn't depend on kafka-clients. With the explicit dependency definition, you are overriding the transitive Kafka dependency from the kafka connector.
[1] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.1/kafka_2.10-0.8.1.pom On Mon, Apr 18, 2016 at 5:38 PM, Robert Schmidtke <ro.schmid...@gmail.com> wrote: > Turns out when I remove the explicit dependency on kafka_2.10 v. 0.8.1, > then the dependencies are properly included. Guess there was a conflict > somehow? I'll need to figure out if the rest of the code is fine with > kafka_2.10 v. 0.8.2.0 as well. > > On Mon, Apr 18, 2016 at 4:32 PM, Robert Schmidtke <ro.schmid...@gmail.com> > wrote: > >> Hi Robert, >> >> thanks for your offer. After playing around a bit I would like to take >> it, if you have the time: >> https://github.com/robert-schmidtke/HiBench/blob/flink-streaming/src/streambench/flinkbench/pom.xml >> >> I would guess the POM is similar to the one in the sample project, yet >> when building it, the jar does not contain all indirect dependencies. >> >> Thanks!! >> >> Robert >> >> On Mon, Apr 18, 2016 at 4:07 PM, Robert Metzger <rmetz...@apache.org> >> wrote: >> >>> If you want you can share the pom of your project privately. >>> >>> On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke < >>> ro.schmid...@gmail.com> wrote: >>> >>>> You're right, it does not. When including it the resulting jar has the >>>> Kafka dependencies bundled. Now it's up to me to figure out the difference >>>> between the sample project and the one I'm working on. >>>> >>>> Thanks! Really quick help. >>>> >>>> Robert >>>> >>>> On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <rmetz...@apache.org> >>>> wrote: >>>> >>>>> Hi, >>>>> the problem with the posted project is that it doesn't have the Flink >>>>> kafka connector as a dependency. >>>>> >>>>> On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke < >>>>> ro.schmid...@gmail.com> wrote: >>>>> >>>>>> Hi Robert, >>>>>> >>>>>> thanks for your hints. I was not sure whether I was building a proper >>>>>> fat jar, as I have not used the Flink Archetype for my project. However, >>>>>> I >>>>>> have set up a sample project at >>>>>> >>>>>> https://github.com/robert-schmidtke/flink-test/ >>>>>> >>>>>> which is nothing more than the Quickstart Archetype plus the >>>>>> instructions to bundle the Kafka dependencies. The resulting jars (mvn >>>>>> clean package -Pbuild-jar and mvn clean package) do not contain the >>>>>> org/apache/kafka/** classes. Can you have a quick look at the pom? >>>>>> However, >>>>>> as I said, it's verbatim Archetype+Flink Docs. >>>>>> >>>>>> Thanks a lot in advance! >>>>>> >>>>>> Robert >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <rmetz...@apache.org >>>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> did you check your user jar if it contains the Kafka classes? >>>>>>> Are you building a fat jar? Are you manually excluding any >>>>>>> dependencies? >>>>>>> >>>>>>> Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in >>>>>>> turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" >>>>>>> dependency also contains the org.apache.kafka.common.Node class (the >>>>>>> LegacyFetcher needs that class). >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom >>>>>>> [2] >>>>>>> https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom >>>>>>> >>>>>>> On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan < >>>>>>> balaji.rajagopa...@olacabs.com> wrote: >>>>>>> >>>>>>>> I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version >>>>>>>> 2.11, was never able to get it working confounded with >>>>>>>> noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2 scala >>>>>>>> version 2.11 things worked for me, if moving to flink 1.0.0 is an >>>>>>>> option >>>>>>>> for you do so. >>>>>>>> >>>>>>>> balaji >>>>>>>> >>>>>>>> On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke < >>>>>>>> ro.schmid...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi everyone, >>>>>>>>> >>>>>>>>> I have a Kafka cluster running on version 0.8.1, hence I'm using >>>>>>>>> the FlinkKafkaConsumer081. When running my program, I saw a >>>>>>>>> NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged >>>>>>>>> my >>>>>>>>> binaries according to >>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, >>>>>>>>> however I'm still seeing the error. >>>>>>>>> >>>>>>>>> I played around a bit and it turns out I have to package >>>>>>>>> kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my >>>>>>>>> program. Is >>>>>>>>> there an error in the documentation or have I not figured out >>>>>>>>> something >>>>>>>>> properly? >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> Robert >>>>>>>>> >>>>>>>>> -- >>>>>>>>> My GPG Key ID: 336E2680 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> My GPG Key ID: 336E2680 >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> My GPG Key ID: 336E2680 >>>> >>> >>> >> >> >> -- >> My GPG Key ID: 336E2680 >> > > > > -- > My GPG Key ID: 336E2680 >
