[
https://issues.apache.org/jira/browse/FLINK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076884#comment-17076884
]
Canbin Zheng edited comment on FLINK-12160 at 4/7/20, 4:13 AM:
---------------------------------------------------------------
Thanks [~fly_in_gis]! I checked again the design doc for the [FLIP-85 Support
Cluster Deploy-Mode|#],] and here is the description for the Kubernetes per-job
cluster:
{panel}
The Kubernetes do not have a default distributed storage and provide public api
to ship files like Yarn local resource. So we could not ship the user jars and
files on the client side to jobmanager and taskmanager. Also it is not a common
way on Kubernetes. Instead, users usually build their jars and files into the
docker image. So when the jobmanager and taskmanager are launched, the users
jars already existed.Even if some users do not want to build the jars into the
image, they could use the initContainer to download the jars from the
storage(http/s3/etc.).All in all, the Kubernetes per-job cluster will only
support cluster deploy-mode.{panel}
If you have new thoughts on the implementation for Kubernetes per-job cluster,
it would be better that we update the doc and further discuss it in the
community.
was (Author: felixzheng):
Thanks [~fly_in_gis]! I checked again the design doc for the [FLIP-85 Support
Cluster
Deploy-Mode|[https://docs.google.com/document/d/1aAwVjdZByA-0CHbgv16Me-vjaaDMCfhX7TzVVTuifYM/edit#],]
and here is the description for the Kubernetes per-job cluster:
{code:java}
The Kubernetes do not have a default distributed storage and provide public api
to ship files like Yarn local resource. So we could not ship the user jars and
files on the client side to jobmanager and taskmanager. Also it is not a common
way on Kubernetes. Instead, users usually build their jars and files into the
docker image. So when the jobmanager and taskmanager are launched, the users
jars already existed.Even if some users do not want to build the jars into the
image, they could use the initContainer to download the jars from the
storage(http/s3/etc.).All in all, the Kubernetes per-job cluster will only
support cluster deploy-mode.
{code}
If you have new thoughts on the implementation for Kubernetes per-job cluster,
it would be better that we update the doc and further discuss it in the
community.
> Support to ship user config files on kubernetes
> -----------------------------------------------
>
> Key: FLINK-12160
> URL: https://issues.apache.org/jira/browse/FLINK-12160
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Major
>
> When deploying flink cluster on Yarn, `yarnShip` cli option could be used to
> ship user config files and jars. The files are registered as Yarn local
> resource which is saved on hdfs.
>
> However, we do not have a default existing dfs. The `ConfigMap` could be used
> to ship small files. The files will be stored in etcd and then mounted to
> jobmanager and taskmanager pod so that they could use it as locally.
> {code:java}
> -kt,--kubernetesship <arg> Ship files in the specified
> directory
> (t for transfer). Only small
> files(<1MB) could be supported.
> {code}
> [https://stackoverflow.com/questions/53012798/kubernetes-configmap-size-limitation]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
