3. Make sense to me. And we could add a new HA solution "StatefulSet + PV +
FileSystem"
at any time if we need in the future.
Since there are no more open questions, I will start the voting now.
Thanks all for your comments and feedback. Feel feel to continue the
discussion if you get
other concer
3. We could avoid force deletions from within Flink. If the user does it,
then we don't give guarantees.
I am fine with your current proposal. +1 for moving forward with it.
Cheers,
Till
On Thu, Oct 1, 2020 at 2:32 AM Yang Wang wrote:
> 2. Yes. This is exactly what I mean. Storing the HA infor
2. Yes. This is exactly what I mean. Storing the HA information relevant to
a specific component in a single ConfigMap and ensuring that “Get(check the
leader)-and-Update(write back to the ConfigMap)” is a transactional
operation. Since we only store the job graph stateHandler(not the real
data) in
Thanks for the clarifications Yang Wang.
2. Keeping the HA information relevant for a component (Dispatcher,
JobManager, ResourceManager) in a single ConfigMap sounds good. We should
check that we don't exceed the 1 MB size limit with this approach though.
The Dispatcher's ConfigMap would then con
Thanks for your explanation. It would be fine if only checking leadership &
actually write information is atomic.
Best,
tison.
Yang Wang 于2020年9月30日周三 下午3:57写道：
> Thanks till and tison for your comments.
>
> @Till Rohrmann
> 1. I am afraid we could not do this if we are going to use fabric8
>
Thanks till and tison for your comments.
@Till Rohrmann
1. I am afraid we could not do this if we are going to use fabric8
Kubernetes client SDK for the leader election. The official Kubernetes Java
client[1] also could not support it. Unless we implement a new
LeaderElector in Flink based on the
Hi,
Generally +1 for a native k8s HA service.
For leader election & publish leader information, there was a discussion[1]
pointed out that since these two actions is NOT atomic, there will be always
edge case where a previous leader overwrite leader information, even with
versioned write. Version
For 1. I was wondering whether we can't write the leader connection
information directly when trying to obtain the leadership (trying to update
the leader key with one's own value)? This might be a little detail, though.
2. Alright, so we are having a similar mechanism as we have in ZooKeeper
with
Hi Till, thanks for your valuable feedback.
1. Yes, leader election and storing leader information will use a same
ConfigMap. When a contender successfully performs a versioned annotation
update operation to the ConfigMap, it means that it has been elected as the
leader. And it will write the lead
Thanks for creating this FLIP Yang Wang. I believe that many of our users
will like a ZooKeeper-less HA setup.
+1 for not separating the leader information and the leader election if
possible. Maybe it is even possible that the contender writes his leader
information directly when trying to obtain
Hi Xintong and Stephan,
Thanks a lot for your attention on this FLIP. I will address the comments
inline.
# Architecture -> One or two ConfigMaps
Both of you are right. One ConfigMap will make the design and
implementation easier. Actually, in my POC codes,
I am using just one ConfigMap(e.g. "k8
This is a very cool feature proposal.
One lesson-learned from the ZooKeeper-based HA is that it is overly
complicated to have the Leader RPC address in a different node than the
LeaderLock. There is extra code needed to make sure these converge and the
can be temporarily out of sync.
A much easie
Thanks for preparing this FLIP, @Yang.
In general, I'm +1 for this new feature. Leveraging Kubernetes's buildtin
ConfigMap for Flink's HA services should significantly reduce the
maintenance overhead compared to deploying a ZK cluster. I think this is an
attractive feature for users.
Concerning t
Hi devs and users,
I would like to start the discussion about FLIP-144[1], which will introduce
a new native high availability service for Kubernetes.
Currently, Flink has provided Zookeeper HA service and been widely used
in production environments. It could be integrated in standalone cluster,
14 matches
Mail list logo
