This is a limitation of the presto version; use
flink-s3-fs-hadoop-1.11.3.jar instead.
On 08/09/2021 20:39, Dhiru wrote:
I copied
FROM flink:1.11.3-scala_2.12-java11 RUN mkdir
./plugins/flink-s3-fs-presto RUN cp
./opt/flink-s3-fs-presto-1.11.3.jar ./plugins/flink-s3-fs-presto/
then started getting this error , trying to run on aws eks and trying
to access s3 bucket
2021-09-08 14:38:10
java.lang.UnsupportedOperationException: This s3 file system
implementation does not support recoverable writers.
at
org.apache.flink.fs.s3.common.FlinkS3FileSystem.createRecoverableWriter(FlinkS3FileSystem.java:136)
at
org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.createRecoverableWriter(PluginFileSystemFactory.java:134)
at
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:260)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$1(StreamTask.java:506)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:526)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.base/java.lang.Thread.run(Unknown Source)
On Wednesday, September 8, 2021, 12:47:10 PM EDT, Chesnay Schepler
<ches...@apache.org> wrote:
you need to put the flink-s3-fs-hadoop/presto jar into a directory
within the plugins directory, for example the final path should look
like this:
/opt/flink/plugins/flink-s3-fs-hadoop/flink-s3-fs-hadoop-1.13.1.jar
Furthermore, you only need either the hadoop or presto jar, _not_ both
of them.
See also:
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins
<https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins>
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/filesystems/plugins/
<https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/filesystems/plugins/>
On 08/09/2021 17:10, Dhiru wrote:
yes I copied to plugin folder but not sure same jar I see in /opt as
well by default
root@d852f125da1f:/opt/flink/plugins# ls
README.txt flink-s3-fs-hadoop-1.13.1.jar metrics-datadog
metrics-influx metrics-prometheus metrics-statsd
external-resource-gpu flink-s3-fs-presto-1.13.1.jar metrics-graphite
metrics-jmx metrics-slf4j
I need help sooner on this
On Wednesday, September 8, 2021, 09:26:46 AM EDT, Dhiru
<userdh...@yahoo.com> <mailto:userdh...@yahoo.com> wrote:
yes I copied to plugin folder but not sure same jar I see in /opt as
well by default
root@d852f125da1f:/opt/flink/plugins# ls
README.txt flink-s3-fs-hadoop-1.13.1.jar metrics-datadog
metrics-influx metrics-prometheus metrics-statsd
external-resource-gpu flink-s3-fs-presto-1.13.1.jar metrics-graphite
metrics-jmx metrics-slf4j
On Wednesday, September 8, 2021, 02:58:38 AM EDT, Martijn Visser
<mart...@ververica.com> <mailto:mart...@ververica.com> wrote:
Hi,
Have you copied the correct JAR [1] to the plugins directory?
Best regards,
Martijn
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/filesystems/s3.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/filesystems/s3.html>
On Wed, 8 Sept 2021 at 04:27, Dhiru <userdh...@yahoo.com
<mailto:userdh...@yahoo.com>> wrote:
Need to configure aws S3 getting this error
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
Could not find a file system implementation for scheme 's3'. The
scheme is directly supported by Flink through the following
plugins: flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure
that each plugin resides within its own subfolder within the
plugins directory. See
https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html
<https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html>
for more information. If you want to use a Hadoop file system for
that scheme, please add the scheme to the configuration
fs.allowed-fallback-filesystems. For a full list of supported file
systems, please see
https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/
<https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/>.
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:473)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:260)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$1(StreamTask.java:506)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:526)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.base/java.lang.Thread.run(Unknown Source)
image used flink:1.11.3-scala_2.12-java11
some of the part which I added for flink-conf.yaml
# REQUIRED: set storage location for job metadata in remote storage
high-availability.storageDir: s3://msc-sandbox-flink/test
state.backend: filesystem
state.checkpoints.dir: s3://msc-sandbox-flink/test
state.backend.fs.checkpointdir: s3://msc-sandbox-flink/test
s3.path.style.access: true
I am trying to deploy jobmanager and taskmanager using eks ,
please let me know if I need to do any more support for adding s3
