Hi, I'd like to submit a job with dependency jars by flink run, but it failed.
Here is the script, /usr/bin/hadoop/software/flink-1.4.2/bin/flink run \ -m yarn-cluster -yn 1 -ys 8 -yjm 2148 -ytm 4096 -ynm jarsTest \ -c StreamExample \ -C file:/home/work/xxx/lib/commons-math3-3.5.jar \ -C file:/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar \ ... xxx-1.0.jar As described in https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/cli.html#usage , "-C" means to provide the dependency jar. After I execute the command, the job succeed to submit, but can not run in flink cluster on yarn. Exceptions is like below: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08 ClassLoader info: URL ClassLoader: file: '/home/work/xxx/lib/commons-math3-3.5.jar' (missing) file: '/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar' (missing) ....... Class not resolvable through given classloader. at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:232) at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:95) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:231) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at java.lang.Thread.run(Thread.java:745) It appears that the two dependency jar cannot be found in TaskManager, so I dig into the source code, from CliFrontend to PackagedProgram to ClusterClient to JobGraph. It seems like the dependency jars is put in classpath and userCodeClassLoader in PackagedProgram, but never upload to the BlobServer in JobGraph where the xxx-1.0.jar is uploaded. Am I missing something? In Flink 1.4.2, dependency jar is not supported? Hope someone can give me some hint. Appreciate it very mush. Yours Sincerely Joshua
