Sounds good. Thank you!
Hao Sun
On Thu, Feb 27, 2020 at 6:52 PM Yang Wang wrote:
> Hi Hao Sun,
>
> I just post the explanation to the user ML so that others could also have
> the same problem.
>
> Gven the job graph is fetched from the jar, do we still need Zookeeper for
>> HA? Maybe we still
Hi Hao Sun,
I just post the explanation to the user ML so that others could also have
the same problem.
Gven the job graph is fetched from the jar, do we still need Zookeeper for
> HA? Maybe we still need it for checkpoint locations?
Yes, we still need the zookeeper(maybe in the future we will
Hi Jin Yi,
For standalone per-job cluster, it is a little different about the
recovery.
Just as you say, the user jar has built in the image, when the JobManager
failed
and relaunched by the K8s, the user `main()` will be executed again to get
the
job graph, not like session cluster to get the job
Hi Yang,
regarding your statement below:
Since you are starting JM/TM with K8s deployment, when they failed new
JM/TM will be created. If you do not set the high
availability configuration, your jobs could recover when TM failed.
However, they could not recover when JM failed. With HA
configured,
I think the only limitation is the disk size of your kubelet machine.
Please remember
to set the "sizeLimit" of your empty dir. Otherwise, your pod may be killed
due to
ephemeral storage is full.
Best,
Yang
M Singh 于2020年2月27日周四 上午8:34写道：
> BTW - Is there any limit to the amount of data that c
BTW - Is there any limit to the amount of data that can be stored on emptyDir
in K8 ?
On Wednesday, February 26, 2020, 07:33:54 PM EST, M Singh
wrote:
Thanks Yang and Arvid for your advice and pointers. Mans
On Wednesday, February 26, 2020, 09:52:26 AM EST, Arvid Heise
wrote:
Thanks Yang and Arvid for your advice and pointers. Mans
On Wednesday, February 26, 2020, 09:52:26 AM EST, Arvid Heise
wrote:
Creds on AWS are typically resolved through roles assigned to K8s pods (for
example with KIAM [1]).
[1] https://github.com/uswitch/kiam
On Tue, Feb 25, 2020 at
Creds on AWS are typically resolved through roles assigned to K8s pods (for
example with KIAM [1]).
[1] https://github.com/uswitch/kiam
On Tue, Feb 25, 2020 at 3:36 AM Yang Wang wrote:
> Hi M Singh,
>
> > Mans - If we use the session based deployment option for K8 - I thought
>> K8 will automat
Hi M Singh,
> Mans - If we use the session based deployment option for K8 - I thought
> K8 will automatically restarts any failed TM or JM.
> In the case of failed TM - the job will probably recover, but in the case
> of failed JM - perhaps we need to resubmit all jobs.
> Let me know if I have mis
Thanks Wang for your detailed answers.
>From what I understand the native_kubernetes also leans towards creating a
>session and submitting a job to it.
Regarding other recommendations, please my inline comments and advice.
On Sunday, February 23, 2020, 10:01:10 PM EST, Yang Wang
wrote:
Hi Singh,
Glad to hear that you are looking to run Flink on the Kubernetes. I am
trying to answer your question based on my limited knowledge and
others could correct me and add some more supplements.
I think the biggest difference between session cluster and per-job cluster
on Kubernetesis the i
Hey Folks:
I am trying to figure out the options for running Flink on Kubernetes and am
trying to find out the pros and cons of running in Flink Session vs Flink
Cluster mode
(https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html#flink-session-cluster-on-kubernete
12 matches
Mail list logo
