Hi, In that case, it won't find the dependencies. Cause I have other dependencies also and what about CEP etc. cause that is not part of flink-dist.
Best Shashank On Wed, Dec 20, 2017 at 3:16 PM, Aljoscha Krettek <aljos...@apache.org> wrote: > Hi, > > That jar file looks like it has too much stuff in there that shouldn't be > there. This can explain the errors you seeing because of classloading > conflicts. > > Could you try not building a fat-jar and have only your code in your jar? > > Best, > Aljoscha > > > On 20. Dec 2017, at 10:15, shashank agarwal <shashank...@gmail.com> wrote: > > One more thing when i submit the job ir start yarn session it prints > following logs : > > Using the result of 'hadoop classpath' to augment the Hadoop classpath: > /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/ > hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0. > 3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/ > usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/ > hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/ > hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6. > 0.3-8/hadoop-mapreduce/.//* > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/opt/flink/lib/ > slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/ > hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > > So i think it's adding Hadoop libs in classpath too cause it's able to > create the checkpointing directories from flink-conf file to HDFS. > > > > > > > > > On Wed, Dec 20, 2017 at 2:31 PM, shashank agarwal <shashank...@gmail.com> > wrote: > >> Hi, >> >> Please find attached list of jar file contents and flink/lib/ contents. I >> have removed my class files list from jar list and I have >> added flink-hadoop-compatibility_2.11-1.4.0.jar later in flink/lib/ but >> no success. >> >> I have tried by removing flink-shaded-hadoop2 from my project but still >> no success. >> >> >> Thanks >> Shashank >> >> >> On Wed, Dec 20, 2017 at 2:14 PM, Aljoscha Krettek <aljos...@apache.org> >> wrote: >> >>> Hi, >>> >>> Could you please list what exactly is in your submitted jar file, for >>> example using "jar tf my-jar-file.jar"? And also what files exactly are in >>> your Flink lib directory. >>> >>> Best, >>> Aljoscha >>> >>> >>> On 19. Dec 2017, at 20:10, shashank agarwal <shashank...@gmail.com> >>> wrote: >>> >>> Hi Timo, >>> >>> I am using Rocksdbstatebackend with hdfs path. I have following flink >>> dependencies in my sbt : >>> >>> "org.slf4j" % "slf4j-log4j12" % "1.7.21", >>> "org.apache.flink" %% "flink-scala" % flinkVersion % "provided", >>> "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % >>> "provided", >>> "org.apache.flink" %% "flink-cep-scala" % flinkVersion, >>> "org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion, >>> "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion, >>> "org.apache.flink" %% "flink-statebackend-rocksdb" % flinkVersion, >>> "org.apache.flink" %% "flink-connector-cassandra" % "1.3.2", >>> "org.apache.flink" % "flink-shaded-hadoop2" % flinkVersion, >>> >>> when i start flink yarn session it's working fine even it's creating >>> flink checkpointing directory and copying libs into hdfs. >>> >>> But when I submit the application to this yarn session it prints >>> following logs : >>> >>> Using the result of 'hadoop classpath' to augment the Hadoop classpath: >>> /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0. >>> >>> <http://2.6.0.0/>3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//* >>> >>> But application fails contuniously with logs which i have sent earlier. >>> >>> >>> >>> I have tried to add flink- hadoop-compability*.jar as suggested by >>> Jorn but it's not working. >>> >>> >>> >>> On Tue, Dec 19, 2017 at 5:08 PM, shashank agarwal <shashank...@gmail.com >>> > wrote: >>> >>>> yes, it's working fine. now not getting compile time error. >>>> >>>> But when i trying to run this on cluster or yarn, getting following >>>> runtime error : >>>> >>>> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not >>>> find a file system implementation for scheme 'hdfs'. The scheme is not >>>> directly supported by Flink and no Hadoop file system to support this >>>> scheme could be loaded. >>>> at >>>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405) >>>> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320) >>>> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) >>>> at >>>> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99) >>>> at >>>> org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277) >>>> at >>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787) >>>> at >>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253) >>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: >>>> Hadoop File System abstraction does not support scheme 'hdfs'. Either no >>>> file system implementation exists for that scheme, or the relevant classes >>>> are missing from the classpath. >>>> at >>>> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:102) >>>> at >>>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401) >>>> ... 12 more >>>> Caused by: java.io.IOException: No FileSystem for scheme: hdfs >>>> at >>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786) >>>> at >>>> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:99) >>>> ... 13 more >>>> >>>> >>>> >>>> >>>> >>>> while submitting job it's printing following logs so i think it's >>>> including hdoop libs : >>>> >>>> >>>> Using the result of 'hadoop classpath' to augment the Hadoop classpath: >>>> /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0. >>>> >>>> <http://2.6.0.0/>3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//* >>>> >>>> >>>> On Fri, Dec 8, 2017 at 9:24 PM, shashank agarwal <shashank...@gmail.com >>>> > wrote: >>>> >>>>> Sure i’ll Try that. Thanks >>>>> >>>>> On Fri, 8 Dec 2017 at 9:18 PM, Stephan Ewen <se...@apache.org> wrote: >>>>> >>>>>> I would recommend to add "flink-shaded-hadoop2". That is a bundle of >>>>>> all Hadoop dependencies used by Flink. >>>>>> >>>>>> >>>>>> On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha Krettek <aljos...@apache.org >>>>>> > wrote: >>>>>> >>>>>>> I see, thanks for letting us know! >>>>>>> >>>>>>> >>>>>>> On 8. Dec 2017, at 15:42, shashank agarwal <shashank...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> I had to include two dependencies. >>>>>>> >>>>>>> hadoop-hdfs (this for HDFS configuration) >>>>>>> hadoop-common (this for Path) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Dec 8, 2017 at 7:38 PM, Aljoscha Krettek < >>>>>>> aljos...@apache.org> wrote: >>>>>>> >>>>>>>> I think hadoop-hdfs might be sufficient. >>>>>>>> >>>>>>>> >>>>>>>> On 8. Dec 2017, at 14:48, shashank agarwal <shashank...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Can you specifically guide which dependencies should I add to >>>>>>>> extend this : >>>>>>>> >>>>>>>> https://github.com/apache/flink/blob/release-1.4.0-rc3/flink >>>>>>>> -connectors/flink-connector-filesystem/src/main/java/org/apa >>>>>>>> che/flink/streaming/connectors/fs/Bucketer.java >>>>>>>> >>>>>>>> is the Hadoop-core is sufficient. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Dec 8, 2017 at 6:58 PM, shashank agarwal < >>>>>>>> shashank...@gmail.com> wrote: >>>>>>>> >>>>>>>>> It's a compilation error. I think I have to include the Hadoop >>>>>>>>> dependencies. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Dec 8, 2017 at 6:54 PM, Aljoscha Krettek < >>>>>>>>> aljos...@apache.org> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Is this a compilation error or at runtime. But in general, yes >>>>>>>>>> you have to include the Hadoop dependencies if they're not there. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Aljoscha >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 8. Dec 2017, at 14:10, shashank agarwal <shashank...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I am trying to test 1.4.0-RC3, Hadoop libraries removed in this >>>>>>>>>> version. Actually, i have created custom Bucketer for the bucketing >>>>>>>>>> sink. >>>>>>>>>> I am extending >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> org.apache.flink.streaming.connectors.fs.bucketing.Bucketer >>>>>>>>>> >>>>>>>>>> in the class, i have to use org.apache.hadoop.fs.Path but as >>>>>>>>>> hadoop libraries removed it's giving error >>>>>>>>>> >>>>>>>>>> "object hadoop is not a member of package org.apache" >>>>>>>>>> >>>>>>>>>> Should i have to include Hadoop client libs in >>>>>>>>>> build.sbt dependencies. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Shashank >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks Regards >>>>>>>>> >>>>>>>>> SHASHANK AGARWAL >>>>>>>>> --- Trying to mobilize the things.... >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks Regards >>>>>>>> >>>>>>>> SHASHANK AGARWAL >>>>>>>> --- Trying to mobilize the things.... >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks Regards >>>>>>> >>>>>>> SHASHANK AGARWAL >>>>>>> --- Trying to mobilize the things.... >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>> Sent from iPhone 5 >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks Regards >>>> >>>> SHASHANK AGARWAL >>>> --- Trying to mobilize the things.... >>>> >>>> >>> >>> >>> -- >>> Thanks Regards >>> >>> SHASHANK AGARWAL >>> --- Trying to mobilize the things.... >>> >>> >>> >> >> >> -- >> Thanks Regards >> >> SHASHANK AGARWAL >> --- Trying to mobilize the things.... >> >> > > > -- > Thanks Regards > > SHASHANK AGARWAL > --- Trying to mobilize the things.... > > > -- Thanks Regards SHASHANK AGARWAL --- Trying to mobilize the things....
