Hmmm, this is caused by a missing dependency (javax.servlet). Could be that this dependency is newly excluded, or that this issue only exists with Hadoop 2.7.0 (HDFS release tests have worked with both newer and older Hadoop versions).
Here is what I think we should do - Can you check whether this works with other Hadoop versions (for example Hadoop 2.7.1 or so?) - To run this in practice, you can also just drop the bundled hadoop dependency and use whatever Hadoop dependency you have already on the cluster (for example via yarn) - Flink needs to adjust some dependencies / exclusions for the pre-bundled Hadoop to make this build error go away In any case, the fact that Hadoop seriously needs the Servlet dependency to initialize HDFS is just ridiculous. Hadoop seriously has the worst dependency management I have ever seen. On Fri, Dec 1, 2017 at 1:41 PM, Vijay Srinivasaraghavan < vijikar...@yahoo.com.invalid> wrote: > Hello, > I am trying to build and run Flink from 1.4.0-rc2 branch with hadoop > binary 2.7.0 compatibility. > Here are the steps I followed to build (I have maven 3.3.9). > ===========================================================================cd > $FLINK_HOMEmvn clean install -DskipTests -Dhadoop.version=2.7.0cd > $FLINK_HOME/flink-distmvn clean install -Dhadoop.version=2.7.0 > Running Flink from $FLINK_HOME/flink-dist/target/ > flink-1.4.0-bin============================================= > ============================= > I am seeing below error error messages from the logs and it suggests > something to do with Hadoop dependency not available? > Could someone please confirm the build steps? > org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not > find a file system implementation for scheme 'hdfs'. The scheme is not > directly supported by Flink and no Hadoop file system to support this > scheme could be loaded. at org.apache.flink.core.fs.FileSystem. > getUnguardedFileSystem(FileSystem.java:403) at org.apache.flink.core.fs. > FileSystem.get(FileSystem.java:318) at > org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.< > init>(FsCheckpointStreamFactory.java:99) at org.apache.flink.runtime. > state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > createCheckpointStreamFactory(StreamTask.java:787) at > org.apache.flink.streaming.api.operators.AbstractStreamOperator. > initializeState(AbstractStreamOperator.java:247) at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > initializeState(StreamTask.java:682) at org.apache.flink.streaming. > runtime.tasks.StreamTask.invoke(StreamTask.java:253) at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at > java.lang.Thread.run(Thread.java:748)Caused by: org.apache.flink.core.fs. > UnsupportedFileSystemSchemeException: Cannot support file system for > 'hdfs' via Hadoop, because Hadoop is not in the classpath, or some classes > are missing from the classpath. at org.apache.flink.runtime.fs. > hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179) at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399) > ... 11 moreCaused by: java.lang.NoClassDefFoundError: Could not > initialize class org.apache.hadoop.hdfs.DFSConfigKeys at > org.apache.hadoop.hdfs.DFSClient$Conf.<init>(DFSClient.java:509) at > org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:637) at > org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize( > DistributedFileSystem.java:149) at org.apache.flink.runtime.fs. > hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159) ... 12 > more2017-12-01 04:28:09,274 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph > - Source: Collection Source -> Map (1/1) ( > 74e8a3f21c86e8dec3c55988fff42e5d) switched from RUNNING to > FAILED.org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Could not find a file system implementation for scheme 'hdfs'. The scheme > is not directly supported by Flink and no Hadoop file system to support > this scheme could be loaded. at org.apache.flink.core.fs.FileSystem. > getUnguardedFileSystem(FileSystem.java:403) at org.apache.flink.core.fs. > FileSystem.get(FileSystem.java:318) at > org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.< > init>(FsCheckpointStreamFactory.java:99) at org.apache.flink.runtime. > state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > createCheckpointStreamFactory(StreamTask.java:787) at > org.apache.flink.streaming.api.operators.AbstractStreamOperator. > initializeState(AbstractStreamOperator.java:247) at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > initializeState(StreamTask.java:682) at org.apache.flink.streaming. > runtime.tasks.StreamTask.invoke(StreamTask.java:253) at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at > java.lang.Thread.run(Thread.java:748)Caused by: org.apache.flink.core.fs. > UnsupportedFileSystemSchemeException: Cannot support file system for > 'hdfs' via Hadoop, because Hadoop is not in the classpath, or some classes > are missing from the classpath. at org.apache.flink.runtime.fs. > hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179) at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399) > ... 11 moreCaused by: java.lang.NoClassDefFoundError: Could not > initialize class org.apache.hadoop.hdfs.DFSConfigKeys at > org.apache.hadoop.hdfs.DFSClient$Conf.<init>(DFSClient.java:509) at > org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:637) at > org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize( > DistributedFileSystem.java:149) at org.apache.flink.runtime.fs. > hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159) ... 12 > more2017-12-01 04:28:09,274 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph > - Source: Collection Source -> Map (1/1) ( > 18fcafbd7bc31a9257de7728b46cee7b) switched from RUNNING to > FAILED.org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Could not find a file system implementation for scheme 'hdfs'. The scheme > is not directly supported by Flink and no Hadoop file system to support > this scheme could be loaded. at org.apache.flink.core.fs.FileSystem. > getUnguardedFileSystem(FileSystem.java:403) at org.apache.flink.core.fs. > FileSystem.get(FileSystem.java:318) at > org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.< > init>(FsCheckpointStreamFactory.java:99) at org.apache.flink.runtime. > state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > createCheckpointStreamFactory(StreamTask.java:787) at > org.apache.flink.streaming.api.operators.AbstractStreamOperator. > initializeState(AbstractStreamOperator.java:247) at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > initializeState(StreamTask.java:682) at org.apache.flink.streaming. > runtime.tasks.StreamTask.invoke(StreamTask.java:253) at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at > java.lang.Thread.run(Thread.java:748)Caused by: org.apache.flink.core.fs. > UnsupportedFileSystemSchemeException: Cannot support file system for > 'hdfs' via Hadoop, because Hadoop is not in the classpath, or some classes > are missing from the classpath. at org.apache.flink.runtime.fs. > hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179) at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399) > ... 11 moreCaused by: java.lang.NoClassDefFoundError: > javax/servlet/Filter at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at > java.net.URLClassLoader.access$100(URLClassLoader.java:73) at > java.net.URLClassLoader$1.run(URLClassLoader.java:368) at > java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security. > AccessController.doPrivileged(Native Method) at java.net.URLClassLoader. > findClass(URLClassLoader.java:361) at java.lang.ClassLoader. > loadClass(ClassLoader.java:424) at sun.misc.Launcher$ > AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader. > loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native > Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at > java.net.URLClassLoader.access$100(URLClassLoader.java:73) at > java.net.URLClassLoader$1.run(URLClassLoader.java:368) at > java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security. > AccessController.doPrivileged(Native Method) at java.net.URLClassLoader. > findClass(URLClassLoader.java:361) at java.lang.ClassLoader. > loadClass(ClassLoader.java:424) at sun.misc.Launcher$ > AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader. > loadClass(ClassLoader.java:357) at org.apache.hadoop.hdfs. > DFSConfigKeys.<clinit>(DFSConfigKeys.java:230) at org.apache.hadoop.hdfs. > DFSClient$Conf.<init>(DFSClient.java:509) at org.apache.hadoop.hdfs. > DFSClient.<init>(DFSClient.java:637) at org.apache.hadoop.hdfs. > DFSClient.<init>(DFSClient.java:619) at org.apache.hadoop.hdfs. > DistributedFileSystem.initialize(DistributedFileSystem.java:149) at > org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159) > ... 12 moreCaused by: java.lang.ClassNotFoundException: > javax.servlet.Filter at java.net.URLClassLoader. > findClass(URLClassLoader.java:381) at java.lang.ClassLoader. > loadClass(ClassLoader.java:424) at sun.misc.Launcher$ > AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader. > loadClass(ClassLoader.java:357) ... 42 more > RegardsVijay
