Re: [DISCUSS] Flink Kubernetes Operator controller flow

Let's keep this discussion for another day so everyone can still chip in. In the meantime I have opened a JIRA ticket for this. Also I invite you to review this PR , it is ready with t

Re: [DISCUSS] Flink Kubernetes Operator controller flow

Hi All! Based on your ideas and suggestions I have made some improvements to the refactor proposal PR: https://github.com/apache/flink-kubernetes-operator/pull/26/commits I have now completely decoupled the observer from the reconciler and the JobManagerDeploymentStatus is recorded in the FlinkDe

Re: [DISCUSS] Flink Kubernetes Operator controller flow

The state machine is going to replace the annoying if-else in the reconciler. It seems to have no conflicts with modular mechanism(aka observer, reconciler, etc.). And we could make them happen step by step. Best, Yang Gyula Fóra 于2022年3月1日周二 13:53写道： > And just one more thing I forgot to add

Re: [DISCUSS] Flink Kubernetes Operator controller flow

And just one more thing I forgot to add: The observer does not have to be a dummy monolithic logic. Different job types will have different state machines with different observe requirements and different observer implementations. I think this is complexly normal. The observer can be aware of the

Re: [DISCUSS] Flink Kubernetes Operator controller flow

@Thomas: Thanks for the input! I completely agree with a well principled state machine based approach in the operator would be the best. You are right that the code contains mostly if then else based logic at the moment as it evolved after initial prototype to cover more scenarios. However I thi

Re: [DISCUSS] Flink Kubernetes Operator controller flow

Thanks for bringing the discussion. It's a good time to revisit this area as the operator implementation becomes more complex. I think the most important part of the design are well defined states and transitions. Unless that can be reasoned about, there will be confusion. For example: 1) For job

Re: [DISCUSS] Flink Kubernetes Operator controller flow

Hi All! Thank you for the feedback, I agree with what has been proposed to include as much as possible in the actual resource status and make the reconciler completely independent from the observer. @Biao Geng: Validation could depend on the current status (depending on how we implement the valid

Re: [DISCUSS] Flink Kubernetes Operator controller flow

Hi Gyula, Thanks for the discussion. It also makes senses to me on the separation of 3 components and Yang's proposal. Just 1 follow-up thought after checking your PR: in the reconcile() method of controller, IIUC, the real flow could be `validate->observe->reconcile->validate->observe->reconcile..

Re: [DISCUSS] Flink Kubernetes Operator controller flow

It is worth looking at the controller code in the spotify operator too: https://github.com/spotify/flink-on-k8s-operator/blob/master/controllers/flinkcluster/flinkcluster_controller.go It is looping in the 'observer phase' until it reaches a stable state, then it performs the necessary changes. B

Re: [DISCUSS] Flink Kubernetes Operator controller flow

Hi, Gyula Thanks for driving this discussion. I second Yang Wang's idea that it's better to make the `validator`, `observer` and `reconciler` self-contained. I also prefer to define the `Observer` as an interface and we could define the statuses that `Observer` will expose. It acts like the o

Re: [DISCUSS] Flink Kubernetes Operator controller flow

Thanks for posting the discussion here. Having the components `validator` `observer` `reconciler` makes lots of sense. And the "Validate -> Observe -> Reconcile" flow seems natural to me. Regarding the implementation in the PR, instead of directly using the observer in the reconciler, I lean to

[DISCUSS] Flink Kubernetes Operator controller flow

Hi All! I would like to start a discussion thread regarding the structure of the Kubernetes Operator controller flo