One more thing when i submit the job ir start yarn session it prints following logs :
Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//* SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/flink/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] So i think it's adding Hadoop libs in classpath too cause it's able to create the checkpointing directories from flink-conf file to HDFS. On Wed, Dec 20, 2017 at 2:31 PM, shashank agarwal <shashank...@gmail.com> wrote: > Hi, > > Please find attached list of jar file contents and flink/lib/ contents. I > have removed my class files list from jar list and I have > added flink-hadoop-compatibility_2.11-1.4.0.jar later in flink/lib/ but > no success. > > I have tried by removing flink-shaded-hadoop2 from my project but still no > success. > > > Thanks > Shashank > > > On Wed, Dec 20, 2017 at 2:14 PM, Aljoscha Krettek <aljos...@apache.org> > wrote: > >> Hi, >> >> Could you please list what exactly is in your submitted jar file, for >> example using "jar tf my-jar-file.jar"? And also what files exactly are in >> your Flink lib directory. >> >> Best, >> Aljoscha >> >> >> On 19. Dec 2017, at 20:10, shashank agarwal <shashank...@gmail.com> >> wrote: >> >> Hi Timo, >> >> I am using Rocksdbstatebackend with hdfs path. I have following flink >> dependencies in my sbt : >> >> "org.slf4j" % "slf4j-log4j12" % "1.7.21", >> "org.apache.flink" %% "flink-scala" % flinkVersion % "provided", >> "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % >> "provided", >> "org.apache.flink" %% "flink-cep-scala" % flinkVersion, >> "org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion, >> "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion, >> "org.apache.flink" %% "flink-statebackend-rocksdb" % flinkVersion, >> "org.apache.flink" %% "flink-connector-cassandra" % "1.3.2", >> "org.apache.flink" % "flink-shaded-hadoop2" % flinkVersion, >> >> when i start flink yarn session it's working fine even it's creating >> flink checkpointing directory and copying libs into hdfs. >> >> But when I submit the application to this yarn session it prints >> following logs : >> >> Using the result of 'hadoop classpath' to augment the Hadoop classpath: >> /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0. >> >> <http://2.6.0.0/>3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//* >> >> But application fails contuniously with logs which i have sent earlier. >> >> >> >> I have tried to add flink- hadoop-compability*.jar as suggested by Jorn >> but it's not working. >> >> >> >> On Tue, Dec 19, 2017 at 5:08 PM, shashank agarwal <shashank...@gmail.com> >> wrote: >> >>> yes, it's working fine. now not getting compile time error. >>> >>> But when i trying to run this on cluster or yarn, getting following >>> runtime error : >>> >>> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not >>> find a file system implementation for scheme 'hdfs'. The scheme is not >>> directly supported by Flink and no Hadoop file system to support this >>> scheme could be loaded. >>> at >>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405) >>> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320) >>> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) >>> at >>> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99) >>> at >>> org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277) >>> at >>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787) >>> at >>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253) >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: >>> Hadoop File System abstraction does not support scheme 'hdfs'. Either no >>> file system implementation exists for that scheme, or the relevant classes >>> are missing from the classpath. >>> at >>> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:102) >>> at >>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401) >>> ... 12 more >>> Caused by: java.io.IOException: No FileSystem for scheme: hdfs >>> at >>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786) >>> at >>> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:99) >>> ... 13 more >>> >>> >>> >>> >>> >>> while submitting job it's printing following logs so i think it's including >>> hdoop libs : >>> >>> >>> Using the result of 'hadoop classpath' to augment the Hadoop classpath: >>> /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0. >>> >>> <http://2.6.0.0/>3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//* >>> >>> >>> On Fri, Dec 8, 2017 at 9:24 PM, shashank agarwal <shashank...@gmail.com> >>> wrote: >>> >>>> Sure i’ll Try that. Thanks >>>> >>>> On Fri, 8 Dec 2017 at 9:18 PM, Stephan Ewen <se...@apache.org> wrote: >>>> >>>>> I would recommend to add "flink-shaded-hadoop2". That is a bundle of >>>>> all Hadoop dependencies used by Flink. >>>>> >>>>> >>>>> On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha Krettek <aljos...@apache.org> >>>>> wrote: >>>>> >>>>>> I see, thanks for letting us know! >>>>>> >>>>>> >>>>>> On 8. Dec 2017, at 15:42, shashank agarwal <shashank...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> I had to include two dependencies. >>>>>> >>>>>> hadoop-hdfs (this for HDFS configuration) >>>>>> hadoop-common (this for Path) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Dec 8, 2017 at 7:38 PM, Aljoscha Krettek <aljos...@apache.org >>>>>> > wrote: >>>>>> >>>>>>> I think hadoop-hdfs might be sufficient. >>>>>>> >>>>>>> >>>>>>> On 8. Dec 2017, at 14:48, shashank agarwal <shashank...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> Can you specifically guide which dependencies should I add to extend >>>>>>> this : >>>>>>> >>>>>>> https://github.com/apache/flink/blob/release-1.4.0-rc3/flink >>>>>>> -connectors/flink-connector-filesystem/src/main/java/org/apa >>>>>>> che/flink/streaming/connectors/fs/Bucketer.java >>>>>>> >>>>>>> is the Hadoop-core is sufficient. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Dec 8, 2017 at 6:58 PM, shashank agarwal < >>>>>>> shashank...@gmail.com> wrote: >>>>>>> >>>>>>>> It's a compilation error. I think I have to include the Hadoop >>>>>>>> dependencies. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Dec 8, 2017 at 6:54 PM, Aljoscha Krettek < >>>>>>>> aljos...@apache.org> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Is this a compilation error or at runtime. But in general, yes you >>>>>>>>> have to include the Hadoop dependencies if they're not there. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Aljoscha >>>>>>>>> >>>>>>>>> >>>>>>>>> On 8. Dec 2017, at 14:10, shashank agarwal <shashank...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I am trying to test 1.4.0-RC3, Hadoop libraries removed in this >>>>>>>>> version. Actually, i have created custom Bucketer for the bucketing >>>>>>>>> sink. >>>>>>>>> I am extending >>>>>>>>> >>>>>>>>> >>>>>>>>> org.apache.flink.streaming.connectors.fs.bucketing.Bucketer >>>>>>>>> >>>>>>>>> in the class, i have to use org.apache.hadoop.fs.Path but as >>>>>>>>> hadoop libraries removed it's giving error >>>>>>>>> >>>>>>>>> "object hadoop is not a member of package org.apache" >>>>>>>>> >>>>>>>>> Should i have to include Hadoop client libs in >>>>>>>>> build.sbt dependencies. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Shashank >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks Regards >>>>>>>> >>>>>>>> SHASHANK AGARWAL >>>>>>>> --- Trying to mobilize the things.... >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks Regards >>>>>>> >>>>>>> SHASHANK AGARWAL >>>>>>> --- Trying to mobilize the things.... >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks Regards >>>>>> >>>>>> SHASHANK AGARWAL >>>>>> --- Trying to mobilize the things.... >>>>>> >>>>>> >>>>>> >>>>> -- >>>> Sent from iPhone 5 >>>> >>> >>> >>> >>> -- >>> Thanks Regards >>> >>> SHASHANK AGARWAL >>> --- Trying to mobilize the things.... >>> >>> >> >> >> -- >> Thanks Regards >> >> SHASHANK AGARWAL >> --- Trying to mobilize the things.... >> >> >> > > > -- > Thanks Regards > > SHASHANK AGARWAL > --- Trying to mobilize the things.... > > -- Thanks Regards SHASHANK AGARWAL --- Trying to mobilize the things....
