[
https://issues.apache.org/jira/browse/FLINK-20505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254403#comment-17254403
]
Xintong Song commented on FLINK-20505:
--------------------------------------
I'm not sure whether we should support individual files for provided lib dirs.
The feature is designed to share the lib/ folder and plugins/ folder via Yarn
distributed cache, so that we don't need to upload them for every application.
The shared contents are cached on Yarn NMs with public visibility, therefore
should not be used for job-specific files. For the lib/ and plugins/ folders,
it does not make much sense to require users to list every single file in the
folders.
That's why I'm asking what contents need to be downloaded from the http paths
in [~zoucao]'s use case.
> Yarn provided lib does not work with http paths.
> ------------------------------------------------
>
> Key: FLINK-20505
> URL: https://issues.apache.org/jira/browse/FLINK-20505
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.12.0, 1.11.2
> Reporter: Xintong Song
> Assignee: zoucao
> Priority: Major
> Labels: pull-request-available
>
> If an http path is used for provided lib, the following exception will be
> thrown on the resource manager side:
> {code:java}
> 2020-12-04 17:01:28.955 ERROR org.apache.flink.yarn.YarnResourceManager -
> Could not start TaskManager in container containerXXXXXX.
> org.apache.flink.util.FlinkException: Error to parse
> YarnLocalResourceDescriptor from YarnLocalResourceDescriptor{key=XXXXX.jar,
> path=https://XXXXXXX.jar, size=-1, modificationTime=0, visibility=APPLICATION}
> at
> org.apache.flink.yarn.YarnLocalResourceDescriptor.fromString(YarnLocalResourceDescriptor.java:99)
> at
> org.apache.flink.yarn.Utils.decodeYarnLocalResourceDescriptorListFromString(Utils.java:721)
> at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:626)
> at
> org.apache.flink.yarn.YarnResourceManager.getOrCreateContainerLaunchContext(YarnResourceManager.java:746)
> at
> org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:726)
> at
> org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:500)
> at
> org.apache.flink.yarn.YarnResourceManager.onContainersOfResourceAllocated(YarnResourceManager.java:455)
> at
> org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:415)
> {code}
> The problem is that, `HttpFileSystem#getFilsStatus` returns file status with
> length `-1`, while `YarnLocalResourceDescriptor` does not recognize the
> negative file length.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
