[
https://issues.apache.org/jira/browse/FLINK-20505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254406#comment-17254406
]
Yang Wang commented on FLINK-20505:
-----------------------------------
A side input of the provided lib feature. When I introduce this feature in
1.11, the main reason is that we want to decrease the unnecessary
uploading/downloading of Flink binaries(e.g. flink-dist*.jar, log4j*.jar,
etc.). So we assume the users could upload the whole Flink binary to HDFS and
set it public read. Then using this feature will make the submission faster.
Benefit from the Yarn public visibility of distributed cache, all the Flink
applications could also benefit from this to avoid unnecessary downloading.
For non-flink binary jars, we do not suggest to configure them via provided
libs. Just as [~ZhenqiuHuang] said, {{yarn.ship-files}} is more appropriate.
Why do we only support directories?
Currently, if you set the provided lib dirs, we assume that the lib and plugins
directories are included. And then Flink client will skip uploading them. If
files are supported, then things become complicated. We need to list all the
files and skip uploading them via comparing with local jars.
> Yarn provided lib does not work with http paths.
> ------------------------------------------------
>
> Key: FLINK-20505
> URL: https://issues.apache.org/jira/browse/FLINK-20505
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.12.0, 1.11.2
> Reporter: Xintong Song
> Assignee: zoucao
> Priority: Major
> Labels: pull-request-available
>
> If an http path is used for provided lib, the following exception will be
> thrown on the resource manager side:
> {code:java}
> 2020-12-04 17:01:28.955 ERROR org.apache.flink.yarn.YarnResourceManager -
> Could not start TaskManager in container containerXXXXXX.
> org.apache.flink.util.FlinkException: Error to parse
> YarnLocalResourceDescriptor from YarnLocalResourceDescriptor{key=XXXXX.jar,
> path=https://XXXXXXX.jar, size=-1, modificationTime=0, visibility=APPLICATION}
> at
> org.apache.flink.yarn.YarnLocalResourceDescriptor.fromString(YarnLocalResourceDescriptor.java:99)
> at
> org.apache.flink.yarn.Utils.decodeYarnLocalResourceDescriptorListFromString(Utils.java:721)
> at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:626)
> at
> org.apache.flink.yarn.YarnResourceManager.getOrCreateContainerLaunchContext(YarnResourceManager.java:746)
> at
> org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:726)
> at
> org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:500)
> at
> org.apache.flink.yarn.YarnResourceManager.onContainersOfResourceAllocated(YarnResourceManager.java:455)
> at
> org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:415)
> {code}
> The problem is that, `HttpFileSystem#getFilsStatus` returns file status with
> length `-1`, while `YarnLocalResourceDescriptor` does not recognize the
> negative file length.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
