I use following dependencies and it works fine . <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>0.9-SNAPSHOT</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> <version>0.9-SNAPSHOT</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-core</artifactId> <version>0.9-SNAPSHOT</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>0.9-SNAPSHOT</version> </dependency> <dependency>
On Mon, Jun 22, 2015 at 10:07 PM, Hawin Jiang <hawin.ji...@gmail.com> wrote: > Hi Marton > > I have to add whole pom.xml file or just only plugin as below. > I saw L286 to L296 are not correct information in pom.xml. > Thanks. > > > > <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId > >maven-assembly-plugin</artifactId> <version>2.4</version> <configuration> > <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </ > descriptorRefs> </configuration> </plugin> > > On Thu, Jun 11, 2015 at 1:43 AM, Márton Balassi <balassi.mar...@gmail.com> > wrote: > >> As for locally I meant the machine that you use for development to see >> whether this works without parallelism. :-) No need to install stuff on >> your Namenode of course. >> Installing Kafka on a machine and having the Kafka Java dependencies >> available for Flink are two very different things. Try adding the following >> [1] to your maven pom. Then execute 'mvn assembly:assembly', this will >> produce a fat jar suffiexed jar-with-dependencies.jar. You should be able >> to run the example form that. >> >> [1] >> https://github.com/mbalassi/flink-dataflow/blob/master/pom.xml#L286-296 >> >> On Thu, Jun 11, 2015 at 10:32 AM, Hawin Jiang <hawin.ji...@gmail.com> >> wrote: >> >>> Dear Marton >>> >>> What do you meaning for locally Eclipse with 'Run'. >>> Do you want to me to run it on Namenode? >>> But my namenode didn't install Kafka. I only installed Kafka on my data >>> node servers. >>> Do I need to install or copy Kafka jar on Namenode? Actually, I don't >>> want to install everything on Name node server. >>> Please advise me. >>> Thanks. >>> >>> >>> My Flink and Hadoop cluster info as below. >>> >>> Flink on NameNode >>> Kafka,Zookeeper and FLink slave1 on Datanode1 >>> Kafka,Zookeeper ,and Flink slave2 on Datanode2 >>> Kafka, Zookeeper and Flink slave3 on Datanode3 >>> >>> >>> >>> On Thu, Jun 11, 2015 at 1:16 AM, Márton Balassi < >>> balassi.mar...@gmail.com> wrote: >>> >>>> Dear Hawin, >>>> >>>> No problem, I am gald that you are giving our Kafka connector a try. :) >>>> The dependencies listed look good. Can you run the example locally from >>>> Eclipse with 'Run'? I suspect that maybe your Flink cluster does not have >>>> the access to the kafka dependency then. >>>> >>>> As a quick test you could copy the kafka jars to the lib folder of your >>>> Flink distribution on all the machines in your cluster. Everything that is >>>> there goes to the classpath of Flink. Another workaround with be to build a >>>> fat jar for your project containing all the dependencies with 'mvn >>>> assembly:assembly'. Neither of these are beautiful but would help tracking >>>> down the root cause. >>>> >>>> On Thu, Jun 11, 2015 at 10:04 AM, Hawin Jiang <hawin.ji...@gmail.com> >>>> wrote: >>>> >>>>> Dear Marton >>>>> >>>>> >>>>> >>>>> Thanks for supporting again. >>>>> >>>>> I am running these examples at the same project and I am using Eclipse >>>>> IDE to submit it to my Flink cluster. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Here is my dependencies >>>>> >>>>> >>>>> ****************************************************************************** >>>>> >>>>> <dependencies> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>*junit*</groupId> >>>>> >>>>> <artifactId>*junit*</artifactId> >>>>> >>>>> <version>4.12</version> >>>>> >>>>> <scope>test</scope> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.flink</groupId> >>>>> >>>>> <artifactId>*flink*-java</artifactId> >>>>> >>>>> <version>0.9.0-milestone-1</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.flink</groupId> >>>>> >>>>> <artifactId>*flink*-clients</artifactId> >>>>> >>>>> <version>0.9.0-milestone-1</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.flink</groupId> >>>>> >>>>> <artifactId>*flink*-streaming-connectors</artifactId> >>>>> >>>>> <version>0.9.0-milestone-1</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.flink</groupId> >>>>> >>>>> <artifactId>*flink*-streaming-core</artifactId> >>>>> >>>>> <version>0.9.0-milestone-1</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.kafka</groupId> >>>>> >>>>> <artifactId>kafka_2.10</artifactId> >>>>> >>>>> <version>0.8.2.1</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.kafka</groupId> >>>>> >>>>> <artifactId>*kafka*-clients</artifactId> >>>>> >>>>> <version>0.8.2.1</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.hadoop</groupId> >>>>> >>>>> <artifactId>*hadoop*-*hdfs*</artifactId> >>>>> >>>>> <version>2.6.0</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.hadoop</groupId> >>>>> >>>>> <artifactId>*hadoop*-*auth*</artifactId> >>>>> >>>>> <version>2.6.0</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.hadoop</groupId> >>>>> >>>>> <artifactId>*hadoop*-common</artifactId> >>>>> >>>>> <version>2.6.0</version> >>>>> >>>>> </dependency> >>>>> >>>>> <dependency> >>>>> >>>>> <groupId>org.apache.hadoop</groupId> >>>>> >>>>> <artifactId>*hadoop*-core</artifactId> >>>>> >>>>> <version>1.2.1</version> >>>>> >>>>> </dependency> >>>>> >>>>> </dependencies> >>>>> >>>>> >>>>> >>>>> >>>>> ***************************************************************************************** >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Best regards >>>>> >>>>> Email: hawin.ji...@gmail.com >>>>> >>>>> >>>>> >>>>> *From:* Márton Balassi [mailto:balassi.mar...@gmail.com] >>>>> *Sent:* Thursday, June 11, 2015 12:58 AM >>>>> *To:* user@flink.apache.org >>>>> *Subject:* Re: Kafka0.8.2.1 + Flink0.9.0 issue >>>>> >>>>> >>>>> >>>>> Dear Hawin, >>>>> >>>>> >>>>> >>>>> This looks like a dependency issue, the java compiler does not find >>>>> the kafka dependency. How are you trying to run this example? Is it from >>>>> an >>>>> IDE or submitting it to a flink cluster with bin/flink run? How do you >>>>> define your dependencies, do you use maven or sbt for instance? >>>>> >>>>> >>>>> >>>>> Best, >>>>> >>>>> >>>>> >>>>> Marton >>>>> >>>>> >>>>> >>>>> On Thu, Jun 11, 2015 at 9:43 AM, Hawin Jiang <hawin.ji...@gmail.com> >>>>> wrote: >>>>> >>>>> >>>>> >>>> >>>> >>> >> >
