Hi,Anubhav Would you like to share the result of `echo $HADOOP_CLASSPATH` and the detailed information after you set up the hadoop classpaht?
Best, Guowei On Wed, Apr 13, 2022 at 4:16 PM Anubhav Nanda <aanubhav.na...@gmail.com> wrote: > Hi Guomei, > > That i already did but still getting the issue > > Regards, > Anubhav > > > > [image: Mailtrack] > <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;> > Sender > notified by > Mailtrack > <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;> > 13/04/22, > 13:46:13 > > On Wed, Apr 13, 2022 at 1:23 PM Guowei Ma <guowei....@gmail.com> wrote: > >> Hi >> I think you need to export HADOOP_CLASSPATH correclty. [1] >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/yarn/#preparation >> Best, >> Guowei >> >> >> On Wed, Apr 13, 2022 at 12:50 PM Anubhav Nanda <aanubhav.na...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I have setup flink 1.13.5 and we are using Hadoop 3.0.0 while we are >>> running simple wordcount example we are getting following error >>> >>> >>> ./flink-1.13.5/bin/flink run flink-1.13.5/examples/batch/WordCount.jar >>> --input hdfs:///tmp/log4j.properties >>> >>> >>> Caused by: org.apache.flink.runtime.JobException: Creating the input >>> splits caused an error: Could not find a file system implementation for >>> scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop >>> file system to support this scheme could be loaded. For a full list of >>> supported file systems, please see >>> https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. >>> >>> at >>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:247) >>> >>> at >>> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:792) >>> >>> at >>> org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:196) >>> >>> at >>> org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.createAndRestoreExecutionGraph(DefaultExecutionGraphFactory.java:107) >>> >>> at >>> org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:342) >>> >>> at >>> org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:190) >>> >>> at >>> org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:122) >>> >>> at >>> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:132) >>> >>> at >>> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:110) >>> >>> at >>> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:340) >>> >>> at >>> org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:317) >>> >>> at >>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:107) >>> >>> at >>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:95) >>> >>> at >>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112) >>> >>> ... 8 more >>> >>> Caused by: >>> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not >>> find a file system implementation for scheme 'hdfs'. The scheme is not >>> directly supported by Flink and no Hadoop file system to support this >>> scheme could be loaded. For a full list of supported file systems, please >>> see >>> https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. >>> >>> at >>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:530) >>> >>> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407) >>> >>> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) >>> >>> at >>> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:599) >>> >>> at >>> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:63) >>> >>> at >>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:234) >>> >>> ... 21 more >>> >>> Caused by: >>> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Cannot >>> support file system for 'hdfs' via Hadoop, because Hadoop is not in the >>> classpath, or some classes are missing from the classpath. >>> >>> at >>> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:189) >>> >>> at >>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526) >>> >>> ... 26 more >>> >>> Caused by: java.lang.NoClassDefFoundError: Could not initialize class >>> org.apache.hadoop.hdfs.HdfsConfiguration >>> >>> at >>> org.apache.flink.runtime.util.HadoopUtils.getHadoopConfiguration(HadoopUtils.java:59) >>> >>> at >>> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:84) >>> >>> ... 27 more >>> >>> >>> >>> >>> >>> >>> [image: Mailtrack] >>> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;> >>> Sender >>> notified by >>> Mailtrack >>> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;> >>> 13/04/22, >>> 10:20:02 >>> >>
