Hi I think you need to export HADOOP_CLASSPATH correclty. [1] [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/yarn/#preparation Best, Guowei
On Wed, Apr 13, 2022 at 12:50 PM Anubhav Nanda <aanubhav.na...@gmail.com> wrote: > Hi, > > I have setup flink 1.13.5 and we are using Hadoop 3.0.0 while we are > running simple wordcount example we are getting following error > > > ./flink-1.13.5/bin/flink run flink-1.13.5/examples/batch/WordCount.jar > --input hdfs:///tmp/log4j.properties > > > Caused by: org.apache.flink.runtime.JobException: Creating the input > splits caused an error: Could not find a file system implementation for > scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop > file system to support this scheme could be loaded. For a full list of > supported file systems, please see > https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. > > at > org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:247) > > at > org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:792) > > at > org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:196) > > at > org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.createAndRestoreExecutionGraph(DefaultExecutionGraphFactory.java:107) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:342) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:190) > > at > org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:122) > > at > org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:132) > > at > org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:110) > > at > org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:340) > > at > org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:317) > > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:107) > > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:95) > > at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112) > > ... 8 more > > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Could not find a file system implementation for scheme 'hdfs'. The scheme > is not directly supported by Flink and no Hadoop file system to support > this scheme could be loaded. For a full list of supported file systems, > please see > https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. > > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:530) > > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407) > > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) > > at > org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:599) > > at > org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:63) > > at > org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:234) > > ... 21 more > > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Cannot support file system for 'hdfs' via Hadoop, because Hadoop is not in > the classpath, or some classes are missing from the classpath. > > at > org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:189) > > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526) > > ... 26 more > > Caused by: java.lang.NoClassDefFoundError: Could not initialize class > org.apache.hadoop.hdfs.HdfsConfiguration > > at > org.apache.flink.runtime.util.HadoopUtils.getHadoopConfiguration(HadoopUtils.java:59) > > at > org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:84) > > ... 27 more > > > > > > > [image: Mailtrack] > <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;> > Sender > notified by > Mailtrack > <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;> > 13/04/22, > 10:20:02 >
