Could you try 1) or 2) and enable debug logging* and share the log with us?
*Usually by adjusting FLINK_HOME/conf/log4j.properties. On Thu, Sep 10, 2020 at 5:38 PM Dan Hill <quietgol...@gmail.com> wrote: > Ah, sorry, it's a copy/paste issue with this email. I've tried both: > 1) using s3a uri with flink-s3-fs-hadoop jar > in /opt/flink/plugins/s3-fs-hadoop. > 2) using s3p uri with flink-s3-fs-presto jar > in /opt/flink/plugins/s3-fs-presto. > 3) loading both 1 and 2 > 4) trying s3 uri. > > When doing 1) > > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Could not find a file system implementation for scheme 's3a'. The scheme is > directly supported by Flink through the following plugin: > flink-s3-fs-hadoop. Please ensure that each plugin resides within its own > subfolder within the plugins directory. See > https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html > for more information. If you want to use a Hadoop file system for that > scheme, please add the scheme to the configuration > fs.allowed-fallback-filesystems. For a full list of supported file systems, > please see > https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/. > > When doing 2) > > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Could not find a file system implementation for scheme 's3p'. The scheme is > directly supported by Flink through the following plugin: > flink-s3-fs-presto. Please ensure that each plugin resides within its own > subfolder within the plugins directory. See > https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html > for more information. If you want to use a Hadoop file system for that > scheme, please add the scheme to the configuration > fs.allowed-fallback-filesystems. For a full list of supported file systems, > please see > https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/. > > etc > > On Thu, Sep 10, 2020 at 8:15 AM Arvid Heise <ar...@ververica.com> wrote: > >> Hi Dan, >> >> s3p is only provided by flink-s3-fs-presto plugin. The plugin you used >> provides s3a. >> (and both provide s3, but it's good to use the more specific prefix). >> >> Best, >> >> Arvid >> >> On Thu, Sep 10, 2020 at 9:24 AM Dan Hill <quietgol...@gmail.com> wrote: >> >>> *Background* >>> I'm converting some prototype Flink v1.11.1 code that uses >>> DataSet/DataTable APIs to use the Table API. >>> >>> *Problem* >>> When switching to using the Table API, my s3 plugins stopped working. I >>> don't know why. I've added the required maven table dependencies to the >>> job. >>> >>> I've tried us moving both the presto and/or the hadoop s3 jars to plugin >>> subfolders. No luck. >>> >>> Any ideas what is wrong? I'm guessing I'm missing something simple. >>> >>> >>> *Error* >>> >>> Caused by: >>> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not >>> find a file system implementation for scheme 's3p'. The scheme is directly >>> supported by Flink through the following plugin: flink-s3-fs-presto. Please >>> ensure that each plugin resides within its own subfolder within the plugins >>> directory. See >>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html >>> for more information. If you want to use a Hadoop file system for that >>> scheme, please add the scheme to the configuration >>> fs.allowed-fallback-filesystems. For a full list of supported file systems, >>> please see >>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/. >>> >>> at >>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:473) >>> >>> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) >>> >>> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:292) >>> >>> at >>> org.apache.flink.table.filesystem.FileSystemTableSink.toStagingPath(FileSystemTableSink.java:232) >>> >>> ... 35 more >>> >>> *ls of plugins directory (same for taskmanager)* >>> >>> kubectl exec pod/flink-jobmanager-0 -- ls -l >>> /opt/flink/plugins/s3-fs-hadoop >>> >>> total 19520 >>> >>> -rw-r--r-- 1 root root 19985452 Sep 10 06:27 >>> flink-s3-fs-hadoop-1.11.1.jar >>> >>> >>> >> >> -- >> >> Arvid Heise | Senior Java Developer >> >> <https://www.ververica.com/> >> >> Follow us @VervericaData >> >> -- >> >> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >> Conference >> >> Stream Processing | Event Driven | Real Time >> >> -- >> >> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >> >> -- >> Ververica GmbH >> Registered at Amtsgericht Charlottenburg: HRB 158244 B >> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji >> (Toni) Cheng >> > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
