Hello,
I am trying to build and run Flink from 1.4.0-rc2 branch with hadoop binary
2.7.0 compatibility.
Here are the steps I followed to build (I have maven 3.3.9).
===========================================================================cd
$FLINK_HOMEmvn clean install -DskipTests -Dhadoop.version=2.7.0cd
$FLINK_HOME/flink-distmvn clean install -Dhadoop.version=2.7.0
Running Flink from
$FLINK_HOME/flink-dist/target/flink-1.4.0-bin==========================================================================
I am seeing below error error messages from the logs and it suggests something
to do with Hadoop dependency not available?
Could someone please confirm the build steps?
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a
file system implementation for scheme 'hdfs'. The scheme is not directly
supported by Flink and no Hadoop file system to support this scheme could be
loaded. at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318) at
org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Cannot support
file system for 'hdfs' via Hadoop, because Hadoop is not in the classpath, or
some classes are missing from the classpath. at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
... 11 moreCaused by: java.lang.NoClassDefFoundError: Could not initialize
class org.apache.hadoop.hdfs.DFSConfigKeys at
org.apache.hadoop.hdfs.DFSClient$Conf.<init>(DFSClient.java:509) at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:637) at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159)
... 12 more2017-12-01 04:28:09,274 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
Collection Source -> Map (1/1) (74e8a3f21c86e8dec3c55988fff42e5d) switched from
RUNNING to
FAILED.org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not
find a file system implementation for scheme 'hdfs'. The scheme is not directly
supported by Flink and no Hadoop file system to support this scheme could be
loaded. at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318) at
org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Cannot support
file system for 'hdfs' via Hadoop, because Hadoop is not in the classpath, or
some classes are missing from the classpath. at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
... 11 moreCaused by: java.lang.NoClassDefFoundError: Could not initialize
class org.apache.hadoop.hdfs.DFSConfigKeys at
org.apache.hadoop.hdfs.DFSClient$Conf.<init>(DFSClient.java:509) at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:637) at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159)
... 12 more2017-12-01 04:28:09,274 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
Collection Source -> Map (1/1) (18fcafbd7bc31a9257de7728b46cee7b) switched from
RUNNING to
FAILED.org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not
find a file system implementation for scheme 'hdfs'. The scheme is not directly
supported by Flink and no Hadoop file system to support this scheme could be
loaded. at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318) at
org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Cannot support
file system for 'hdfs' via Hadoop, because Hadoop is not in the classpath, or
some classes are missing from the classpath. at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
... 11 moreCaused by: java.lang.NoClassDefFoundError: javax/servlet/Filter at
java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:763) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at
java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at
java.net.URLClassLoader.access$100(URLClassLoader.java:73) at
java.net.URLClassLoader$1.run(URLClassLoader.java:368) at
java.net.URLClassLoader$1.run(URLClassLoader.java:362) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:361) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:763) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at
java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at
java.net.URLClassLoader.access$100(URLClassLoader.java:73) at
java.net.URLClassLoader$1.run(URLClassLoader.java:368) at
java.net.URLClassLoader$1.run(URLClassLoader.java:362) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:361) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
org.apache.hadoop.hdfs.DFSConfigKeys.<clinit>(DFSConfigKeys.java:230) at
org.apache.hadoop.hdfs.DFSClient$Conf.<init>(DFSClient.java:509) at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:637) at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159)
... 12 moreCaused by: java.lang.ClassNotFoundException: javax.servlet.Filter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 42 more
RegardsVijay
