[
https://issues.apache.org/jira/browse/FLINK-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258161#comment-17258161
]
Xintong Song commented on FLINK-20811:
--------------------------------------
Thanks for the reminder, [~nicholasjiang]. I wasn't aware of FLINK-20681.
[~trohrmann],
I'm wondering whether it's possible to support the ship files feature for both
Yarn/K8s with one common implementation. I don't have an answer atm, but if the
answer is yes, we may not need to implement the feature twice.
On yarn, we leverage the hdfs for registering local resources, so yarn will
localize the ship files for us. In K8s, seems there isn't a role equivalent to
the hdfs, we might need to support implement the ship files feature ourselves.
The question is, if we can support this feature without depending on
Kubernetes, shouldn't it also work for the Yarn integration?
Again, I do not have a thoughtful idea atm. Just want to bring up this
possibility.
> Support HTTP paths for yarn ship files/archives
> -----------------------------------------------
>
> Key: FLINK-20811
> URL: https://issues.apache.org/jira/browse/FLINK-20811
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
> Reporter: Xintong Song
> Priority: Major
>
> Flink's Yarn integration supports shipping workload-specific local
> files/directories/archives to the Yarn cluster.
> As discussed in FLINK-20505, it would be helpful to support directly
> downloading contents from HTTP paths to the Yarn cluster, so that users won't
> need to first download the contents locally and then upload it to the Yarn
> cluster.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
