[
https://issues.apache.org/jira/browse/FLINK-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259561#comment-17259561
]
Till Rohrmann commented on FLINK-20811:
---------------------------------------
I could see it work if we allow users to specify arbitrary URIs which are then
localized by the {{ClusterEntrypoint}} by using Flink's FileSystems or directly
downloading from HTTP, for example. What this would mean, though, is that we
are rebuilding Yarn's {{LocalResource}} mechanism a bit. If Flink's Yarn
integration would use this new mechanism, then we would probably lose the
functionality around {{LocalResourceVisibility}} which allows to easily share
objects between application without caching them multiple times.
Also, we will create multiple ways how to achieve the same thing. For K8s, one
can then either ship the files or build a Docker image.
> Support HTTP paths for yarn ship files/archives
> -----------------------------------------------
>
> Key: FLINK-20811
> URL: https://issues.apache.org/jira/browse/FLINK-20811
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
> Reporter: Xintong Song
> Priority: Major
>
> Flink's Yarn integration supports shipping workload-specific local
> files/directories/archives to the Yarn cluster.
> As discussed in FLINK-20505, it would be helpful to support directly
> downloading contents from HTTP paths to the Yarn cluster, so that users won't
> need to first download the contents locally and then upload it to the Yarn
> cluster.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
