Re: getting an error when configuring state backend to hdfs

[Chesnay Schepler]

flink-hadoop-fs should be in /lib

On 19.12.2018 16:44, Avi Levi wrote:

Hi Chesnay,

What do you mean? I am creating a fat jar with all dependencies (using sbt assembly). which jar I should place in the /lib directory ?

On Wed, Dec 19, 2018 at 4:44 PM Chesnay Schepler <ches...@apache.org <mailto:ches...@apache.org>> wrote:

    Are you including the filesystems in your jar? Filesystem jars
    must be placed in the /lib directory of the flink distribution.

    On 19.12.2018 15:03, Avi Levi wrote:

    Hi,
    I am trying to set the backend state to hdfs
    /val stateUri = "hdfs/path_to_dir"/
    /val backend: RocksDBStateBackend = new
    RocksDBStateBackend(stateUri, true)/
    /env.setStateBackend(backend)
    /
    /
    /
    I am running with flink 1.7.0 with the following dependencies
    (tried them with different combinations)  :
    /"org.apache.flink"   %% "flink-connector-filesystem"    % flinkV/
    /"org.apache.flink"   % "flink-hadoop-fs"    % flinkV/
    /"org.apache.hadoop"  % "hadoop-hdfs"    % hadoopVersion/
    /"org.apache.hadoop"  % "hadoop-common"      % hadoopVersion

    /
    *however when running the jar I am getting this error:*
    *
    *
    /Caused by:
    org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
    Could not find a file system implementation for scheme 'hdfs'.
    The scheme is not directly supported by Flink and no Hadoop file
    system to support this scheme could be loaded./
    /at
    
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)/
    /at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)/
    /at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)/
    /at
    
org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58)/
    /at
    
org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444)/
    /at
    
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createCheckpointStorage(RocksDBStateBackend.java:407)/
    /at
    
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249)/
    /... 17 more/
    /Caused by:
    org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
    Hadoop is not in the classpath/dependencies./
    /at
    
org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)/
    /at
    
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)/
    /... 23 more/

    any help will be greatly appreciated