[
https://issues.apache.org/jira/browse/FLINK-20505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254508#comment-17254508
]
zoucao commented on FLINK-20505:
--------------------------------
hi [~xintongsong],
we use thrift to deserialize (and serialize) our records recieved from(sent to)
MQ. Each thrift table has it's unique jar and the jar only contains it's own
information, so that we need to upload the thrift jar to yarn. In the previous
version, for example 1.9, 1.10, we add a new parameter in `YarnConfigOptions`,
whose name is `io.extra-file`. When submit a Flink job to yarn, we will add
http path in `io.extra-file`, the path is got from artifactory, such that
flink can download it directly. When a http path is added, we will add it to
envShipResourceList, then taskManager can get and load it to classloader. We
designed a thrift format, and used
Thread.currentThread().getContextClassLoader().loadClass(className) to get
thrift class.
hi [~ZhenqiuHuang],
thanks for your advise, I think you are right, may be `yarn.ship-files` is the
right choice for me, after looking the code around `yarn.ship-files` roughly.
At the begining of reporting this jira, I just want let
YarnLocalResourceDescriptor support negative length, http paths could be
supported in provided libs will be better. But now, I don‘t think it is
nessary, even inappropriate, after [~fly_in_gis] sharing his view. Although
supporting http paths in provided libs makes Flink more supportive, things will
become complicated, just as [~fly_in_gis] said.
> Yarn provided lib does not work with http paths.
> ------------------------------------------------
>
> Key: FLINK-20505
> URL: https://issues.apache.org/jira/browse/FLINK-20505
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.12.0, 1.11.2
> Reporter: Xintong Song
> Assignee: zoucao
> Priority: Major
> Labels: pull-request-available
>
> If an http path is used for provided lib, the following exception will be
> thrown on the resource manager side:
> {code:java}
> 2020-12-04 17:01:28.955 ERROR org.apache.flink.yarn.YarnResourceManager -
> Could not start TaskManager in container containerXXXXXX.
> org.apache.flink.util.FlinkException: Error to parse
> YarnLocalResourceDescriptor from YarnLocalResourceDescriptor{key=XXXXX.jar,
> path=https://XXXXXXX.jar, size=-1, modificationTime=0, visibility=APPLICATION}
> at
> org.apache.flink.yarn.YarnLocalResourceDescriptor.fromString(YarnLocalResourceDescriptor.java:99)
> at
> org.apache.flink.yarn.Utils.decodeYarnLocalResourceDescriptorListFromString(Utils.java:721)
> at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:626)
> at
> org.apache.flink.yarn.YarnResourceManager.getOrCreateContainerLaunchContext(YarnResourceManager.java:746)
> at
> org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:726)
> at
> org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:500)
> at
> org.apache.flink.yarn.YarnResourceManager.onContainersOfResourceAllocated(YarnResourceManager.java:455)
> at
> org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:415)
> {code}
> The problem is that, `HttpFileSystem#getFilsStatus` returns file status with
> length `-1`, while `YarnLocalResourceDescriptor` does not recognize the
> negative file length.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
