Thank you Arvid and Yang!

________________________________
From: Yang Wang <danrtsey...@gmail.com>
Sent: Thursday, August 13, 2020 8:09:13 PM
To: Arvid Heise <ar...@ververica.com>
Cc: Alexey Trenikhun <yen...@msn.com>; user <user@flink.apache.org>
Subject: Re: k8s job cluster using StatefulSet

Hi Alexey,

Actually, StatefulSets could also be used to start the JobManager and 
TaskManager.

So why do we suggest to use Deployment in the Flink documentation?
* StatefulSets requires the user to have persistent volume in the K8s cluster. 
However, it is not always true,
  especially for the unmanaged(self-build) K8s cluster.
* Flink uses Zookeeper and distributed storage(S3, GFS, etc.) to process the 
fault tolerance. If you start multiple
  JobManagers, the leader election and leader retrieval will be done via 
Zookeeper. Also the meta information will
   be stored in the Zookeeper. So it is unnecessary to use StatefulSet to do 
more things.
* The local data of JobManager and TaskManager is ephemeral. It could be 
discarded after crashed.


Best,
Yang




Arvid Heise <ar...@ververica.com<mailto:ar...@ververica.com>> 于2020年8月13日周四 
下午4:38写道:
Hi Alexey,

I don't see any issue in using stateful sets immediately.

I'd recommend using one of the K8s operators or Ververica's community edition 
[1] though if you start with a new setup as they may solve even more issues 
that you might experience in the future.

[1] https://www.ververica.com/getting-started

On Mon, Aug 10, 2020 at 11:22 PM Alexey Trenikhun 
<yen...@msn.com<mailto:yen...@msn.com>> wrote:
Hello,
Flink documentation suggests to use Deployments to deploy JM and TM for 
kubernetes job cluster. Is any known potential issues with using StatefulSets 
instead, seems StatefullSet provides uniqueness for JM during upgrade/rollback, 
while with Deployments could be multiple JM pods (e.g.1 terminating and 1 
running)

Thanks,
Alexey


--

Arvid Heise | Senior Java Developer

[https://lh5.googleusercontent.com/ODbO0aq1IqKMfuoy_pw2YH8r6dqDRTq37rg3ytg11FCGJx12jJ1ff_SANPBxTHzSJTUQY9JLuoXq4NB7Om7j6Vq1lg6jIOKz8S5g2VKDGwicbj5fbY09PVb6mD5TdRuWEUvEMZTG]<https://www.ververica.com/>


Follow us @VervericaData

--

Join Flink Forward<https://flink-forward.org/> - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) 
Cheng

Reply via email to