Hi Mike, I would recommend you to build a "fat jar" containing your application code and all required dependencies.
On Tue, May 23, 2017 at 10:33 AM, Mikhail Pryakhin <m.prya...@cleverdata.ru> wrote: > Hi Robert! > > You were right.. it was my fault, I didn’t copy all required dependencies > for the flink-connector-kafka. That made impossible to load the class. > > Everything works as expected but only if I put required dependencies > inside the flink lib folder, but when dependencies are located in a > different folder which is passed in the yarnship command line option then > they seem not to be put on the TM's classpath, and the job fails with > ClassNotFoundException. > Is there anything else I need to provide to make libraries in that folder > to be on the TM's classpath? > > Many thanks, > > —— > > Mike Pryakhin > > > On 22 May 2017, at 23:06, Mikhail Pryakhin <m.prya...@cleverdata.ru> > wrote: > > Hi Robert! > Thanks a lot for your reply! > > >Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar > contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 > ? > > The jar does contain the class org/apache/flink/streaming/ > connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file > is listed below) > > > Also, were there any previous Kafka09 related exceptions in the log?? > No, it was the very first exception… You can find log file attached. > > > the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following: > > root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar > Archive: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar > Length Date Time Name > --------- ---------- ----- ---- > 0 2017-04-11 01:59 META-INF/ > 443 2017-04-11 01:59 META-INF/MANIFEST.MF > 1451 2017-04-11 01:59 META-INF/DEPENDENCIES > 11358 2017-04-11 01:59 META-INF/LICENSE > 182 2017-04-11 01:59 META-INF/NOTICE > 0 2017-04-11 01:59 org/ > 0 2017-04-11 01:59 org/apache/ > 0 2017-04-11 01:59 org/apache/flink/ > 0 2017-04-11 01:59 org/apache/flink/streaming/ > 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/ > 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/ > 11554 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/ > FlinkKafkaConsumer09.class > 2603 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/ > Kafka09JsonTableSink.class > 4463 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/ > FlinkKafkaProducer09.class > 2693 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/ > Kafka09TableSource.class > 0 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/ > 1125 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/KafkaConsumerCallBridge.class > 570 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/Handover$WakeupException.class > 10268 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/KafkaConsumerThread.class > 570 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/Handover$ClosedException.class > 3018 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/Handover.class > 11854 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/Kafka09Fetcher.class > 2182 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class > 311 2017-04-11 01:59 org/apache/flink/streaming/ > connectors/kafka/internal/KafkaConsumerThread$1.class > 2329 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/ > Kafka09JsonTableSource.class > 0 2017-04-11 01:59 META-INF/maven/ > 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/ > 0 2017-04-11 01:59 META-INF/maven/org.apache. > flink/flink-connector-kafka-0.9_2.11/ > 6039 2017-04-11 01:59 META-INF/maven/org.apache. > flink/flink-connector-kafka-0.9_2.11/pom.xml > 131 2017-04-11 01:59 META-INF/maven/org.apache. > flink/flink-connector-kafka-0.9_2.11/pom.properties > 0 2017-04-11 01:59 META-INF/maven/org.apache. > flink/force-shading/ > 3285 2017-04-11 01:59 META-INF/maven/org.apache. > flink/force-shading/pom.xml > 114 2017-04-11 01:59 META-INF/maven/org.apache. > flink/force-shading/pom.properties > --------- ------- > 76543 33 files > > > > > <flink-root-client-dmpkit-dev-dn1.log> > > > ------ > Mike Pryakhin > > > > On 22 May 2017, at 22:14, Robert Metzger <rmetz...@apache.org> wrote: > > Hi, > > this issue is unexpected :) Can you double check if the > job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains > org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ? > Also, were there any previous Kafka09 related exceptions in the log?? > > From this SO answer, it seems that this is not really the classical > classNotFoundException, but a bit differenT: https:// > stackoverflow.com/a/5756989/568695 > > On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.prya...@cleverdata.ru > > wrote: > >> Hi all! >> >> I'm playing with flink streaming job on yarn cluster. The job consumes >> events from kafka and prints them to the standard out. >> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is >> passed via the --yarnship option. >> Here is the way I run the job: >> >> export HADOOP_USER_NAME=hdfs; \ >> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \ >> /opt/flink-1.2.1/bin/flink run \ >> -yst \ >> -yt /home/user/job-libs \ >> -m yarn-cluster \ >> -yn 3 \ >> -c com.flink.Test \ >> flink-test_2.11-1.0.0-SNAPSHOT.jar >> >> Finally the job fails complaing that it can't find the class: >> java.lang.NoClassDefFoundError: org/apache/flink/streaming/con >> nectors/kafka/FlinkKafkaConsumer09 >> >> I looked through the jobmanager.log and found that the >> flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the >> classpath: >> >> 2017-05-22 17:41:59,637 INFO >> org.apache.flink.yarn.YarnApplicationMasterRunner >> - Classpath: >> job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another >> jars...> >> >> Could please help to figure out why the class >> org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't >> be loaded inspite of the jar containing this class was added to the >> classpath >> >> ---- >> Mike Pryakhin > > > > >
