[
https://issues.apache.org/jira/browse/FLINK-32315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893734#comment-17893734
]
Ferenc Csaky edited comment on FLINK-32315 at 10/29/24 8:31 AM:
----------------------------------------------------------------
When I developed this, the "s3" related config options required to be added to
the {{flink-conf.yaml}}, cause those props are not picked up if they given as a
dynamic param in the {{flink run}} command. For Flink 1.20 it was already
renamed {{config.yaml}}, but that should not matter for this particular case I
believe.
I have the following gist about how I was setting up Minio, which might be
outdated, cause it uses the latest img, but still could be helpful:
[https://gist.github.com/ferenc-csaky/fd7fee71d89cd389cac2da4a4471ab65]
was (Author: JIRAUSER306586):
When I developed this, the "s3" related config options required to be added to
the {{{}flink-conf.yaml{}}}, cause those props are not picked up if they given
as a dynamic param during. For Flink 1.20 it was already renamed
{{{}config.yaml{}}}, but that should not matter for this particular case I
believe.
I have the following gist about how I was setting up Minio, which might be
outdated, cause it uses the latest img, but still could be helpful:
[https://gist.github.com/ferenc-csaky/fd7fee71d89cd389cac2da4a4471ab65]
> Support local file upload in K8s mode
> -------------------------------------
>
> Key: FLINK-32315
> URL: https://issues.apache.org/jira/browse/FLINK-32315
> Project: Flink
> Issue Type: New Feature
> Components: Client / Job Submission, Deployment / Kubernetes
> Reporter: Paul Lin
> Assignee: Ferenc Csaky
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Attachments: image-2024-10-29-11-32-25-979.png
>
>
> Currently, Flink assumes all resources are locally accessible in the pods,
> which requires users to prepare the resources by mounting storages,
> downloading resources with init containers, or rebuilding images for each
> execution.
> We could make things much easier by introducing a built-in file distribution
> mechanism based on Flink-supported filesystems. It's implemented in two steps:
>
> 1. KubernetesClusterDescripter uploads all local resources to remote storage
> via Flink filesystem (skips if the resources are already remote).
> 2. KubernetesApplicationClusterEntrypoint and KubernetesTaskExecutorRunner
> download the resources and put them in the classpath during startup.
>
> The 2nd step is mostly done by
> [FLINK-28915|https://issues.apache.org/jira/browse/FLINK-28915], thus this
> issue is focused on the upload part.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
