[
https://issues.apache.org/jira/browse/FLINK-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258223#comment-17258223
]
Yang Wang commented on FLINK-20811:
-----------------------------------
I want to share more information about shipping user jars/artifacts in native
K8s integration. Currently, we require that user jars/artifacts are bundled in
the docker image. I think it is also the typical mechanism in container world.
However, it is not very convenient since we need to build different images for
each Flink application.
The first solution is using init container(could be done via pod template[1]).
Then we could download the user jars/artifacts from remote storage(e.g. http,
S3, HDFS, FTP, etc.).
The second solution could be leveraging the Flink filesystem plugin
implementation. For example, we could have the config option
{{kubernetes.ship-files}}. To enable the shipping, users also need to configure
a valid distributed storage(e.g. S3, HDFS) which is directly supported by Flink
filesystem. First, Flink client will upload the user jars/artifacts to
distributed storage. Second, once the JobManager is launched, it will fetch the
user jars/artifacts. Third, the embedded client will run the user jar and
submit the Flink job. This mean the user jars/artifacts will be uploaded to
blob server and then fetched by TaskManager.
[1]. https://issues.apache.org/jira/browse/FLINK-15656
> Support HTTP paths for yarn ship files/archives
> -----------------------------------------------
>
> Key: FLINK-20811
> URL: https://issues.apache.org/jira/browse/FLINK-20811
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
> Reporter: Xintong Song
> Priority: Major
>
> Flink's Yarn integration supports shipping workload-specific local
> files/directories/archives to the Yarn cluster.
> As discussed in FLINK-20505, it would be helpful to support directly
> downloading contents from HTTP paths to the Yarn cluster, so that users won't
> need to first download the contents locally and then upload it to the Yarn
> cluster.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
