Hi Marton I have to add whole pom.xml file or just only plugin as below. I saw L286 to L296 are not correct information in pom.xml. Thanks.
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId >maven-assembly-plugin</artifactId> <version>2.4</version> <configuration> < descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </ descriptorRefs> </configuration> </plugin> On Thu, Jun 11, 2015 at 1:43 AM, Márton Balassi <balassi.mar...@gmail.com> wrote: > As for locally I meant the machine that you use for development to see > whether this works without parallelism. :-) No need to install stuff on > your Namenode of course. > Installing Kafka on a machine and having the Kafka Java dependencies > available for Flink are two very different things. Try adding the following > [1] to your maven pom. Then execute 'mvn assembly:assembly', this will > produce a fat jar suffiexed jar-with-dependencies.jar. You should be able > to run the example form that. > > [1] > https://github.com/mbalassi/flink-dataflow/blob/master/pom.xml#L286-296 > > On Thu, Jun 11, 2015 at 10:32 AM, Hawin Jiang <hawin.ji...@gmail.com> > wrote: > >> Dear Marton >> >> What do you meaning for locally Eclipse with 'Run'. >> Do you want to me to run it on Namenode? >> But my namenode didn't install Kafka. I only installed Kafka on my data >> node servers. >> Do I need to install or copy Kafka jar on Namenode? Actually, I don't >> want to install everything on Name node server. >> Please advise me. >> Thanks. >> >> >> My Flink and Hadoop cluster info as below. >> >> Flink on NameNode >> Kafka,Zookeeper and FLink slave1 on Datanode1 >> Kafka,Zookeeper ,and Flink slave2 on Datanode2 >> Kafka, Zookeeper and Flink slave3 on Datanode3 >> >> >> >> On Thu, Jun 11, 2015 at 1:16 AM, Márton Balassi <balassi.mar...@gmail.com >> > wrote: >> >>> Dear Hawin, >>> >>> No problem, I am gald that you are giving our Kafka connector a try. :) >>> The dependencies listed look good. Can you run the example locally from >>> Eclipse with 'Run'? I suspect that maybe your Flink cluster does not have >>> the access to the kafka dependency then. >>> >>> As a quick test you could copy the kafka jars to the lib folder of your >>> Flink distribution on all the machines in your cluster. Everything that is >>> there goes to the classpath of Flink. Another workaround with be to build a >>> fat jar for your project containing all the dependencies with 'mvn >>> assembly:assembly'. Neither of these are beautiful but would help tracking >>> down the root cause. >>> >>> On Thu, Jun 11, 2015 at 10:04 AM, Hawin Jiang <hawin.ji...@gmail.com> >>> wrote: >>> >>>> Dear Marton >>>> >>>> >>>> >>>> Thanks for supporting again. >>>> >>>> I am running these examples at the same project and I am using Eclipse >>>> IDE to submit it to my Flink cluster. >>>> >>>> >>>> >>>> >>>> >>>> Here is my dependencies >>>> >>>> >>>> ****************************************************************************** >>>> >>>> <dependencies> >>>> >>>> <dependency> >>>> >>>> <groupId>*junit*</groupId> >>>> >>>> <artifactId>*junit*</artifactId> >>>> >>>> <version>4.12</version> >>>> >>>> <scope>test</scope> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.flink</groupId> >>>> >>>> <artifactId>*flink*-java</artifactId> >>>> >>>> <version>0.9.0-milestone-1</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.flink</groupId> >>>> >>>> <artifactId>*flink*-clients</artifactId> >>>> >>>> <version>0.9.0-milestone-1</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.flink</groupId> >>>> >>>> <artifactId>*flink*-streaming-connectors</artifactId> >>>> >>>> <version>0.9.0-milestone-1</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.flink</groupId> >>>> >>>> <artifactId>*flink*-streaming-core</artifactId> >>>> >>>> <version>0.9.0-milestone-1</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.kafka</groupId> >>>> >>>> <artifactId>kafka_2.10</artifactId> >>>> >>>> <version>0.8.2.1</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.kafka</groupId> >>>> >>>> <artifactId>*kafka*-clients</artifactId> >>>> >>>> <version>0.8.2.1</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.hadoop</groupId> >>>> >>>> <artifactId>*hadoop*-*hdfs*</artifactId> >>>> >>>> <version>2.6.0</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.hadoop</groupId> >>>> >>>> <artifactId>*hadoop*-*auth*</artifactId> >>>> >>>> <version>2.6.0</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.hadoop</groupId> >>>> >>>> <artifactId>*hadoop*-common</artifactId> >>>> >>>> <version>2.6.0</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.hadoop</groupId> >>>> >>>> <artifactId>*hadoop*-core</artifactId> >>>> >>>> <version>1.2.1</version> >>>> >>>> </dependency> >>>> >>>> </dependencies> >>>> >>>> >>>> >>>> >>>> ***************************************************************************************** >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Best regards >>>> >>>> Email: hawin.ji...@gmail.com >>>> >>>> >>>> >>>> *From:* Márton Balassi [mailto:balassi.mar...@gmail.com] >>>> *Sent:* Thursday, June 11, 2015 12:58 AM >>>> *To:* user@flink.apache.org >>>> *Subject:* Re: Kafka0.8.2.1 + Flink0.9.0 issue >>>> >>>> >>>> >>>> Dear Hawin, >>>> >>>> >>>> >>>> This looks like a dependency issue, the java compiler does not find the >>>> kafka dependency. How are you trying to run this example? Is it from an IDE >>>> or submitting it to a flink cluster with bin/flink run? How do you define >>>> your dependencies, do you use maven or sbt for instance? >>>> >>>> >>>> >>>> Best, >>>> >>>> >>>> >>>> Marton >>>> >>>> >>>> >>>> On Thu, Jun 11, 2015 at 9:43 AM, Hawin Jiang <hawin.ji...@gmail.com> >>>> wrote: >>>> >>>> >>>> >>> >>> >> >
