ruanwenjun opened a new issue, #7634: URL: https://github.com/apache/kyuubi/issues/7634
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched the existing issues and found no issue covering coordinated cleanup across multiple Kyuubi servers. ### Describe the feature When multiple Kyuubi server instances monitor the same Kubernetes context and namespace, each instance starts a Pod informer and maintains its own terminated-application cleanup cache. After a Spark Driver Pod reaches a terminal state, every server instance that observed the event independently expires its local cache entry and calls the Kubernetes DELETE API: https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala#L145-L210 This has two side effects: 1. For one terminated Driver Pod, the API server may receive one DELETE request from every Kyuubi server instance. Only the first request deletes the Pod; later requests normally observe that it is already absent. 2. An empty result from Fabric8 DELETE is currently logged as "Failed to delete pod", although a named-resource HTTP 404 is converted by Fabric8 into an empty result. Real authorization, throttling, network, or server failures are therefore difficult to distinguish from an idempotent no-op. The DELETE event is also passed to `markApplicationTerminated`: https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala#L400-L434 The cleanup state does not record that the Pod is already deleting or absent. An existing cleanup trigger is not canceled, and a trigger that has already expired may be created again, causing another redundant DELETE attempt later. The cleanup responsibility should be coordinated independently from application-state observation. ### Motivation The number of mutating Kubernetes API requests for terminated Driver Pods currently grows with the number of Kyuubi server replicas. The resulting warning logs are also not actionable because they conflate an already-absent Pod with a real DELETE failure. A coordinated cleaner would reduce DELETE requests and preserve useful failure diagnostics without changing how individual Kyuubi servers monitor applications. This is different from #7293, which is about Kubernetes clients and informers not being initialized in some deployments. ### Describe the solution A possible design is: 1. Elect one cleanup leader for each `KubernetesInfo(context, namespace)` through the existing Kyuubi HA backend (ZooKeeper or etcd). 2. Keep Pod informers on all Kyuubi servers for application-state tracking, audit logging, metadata persistence, cancellation handling, and local `getApplicationInfoByTag` calls. Leadership gates only the cleanup responsibility. 3. Only the cleanup leader schedules or executes deletion of terminated Driver Pods for its context and namespace. 4. When leadership is acquired, list and reconcile existing terminated Driver Pods so that cleanup continues after the previous leader fails or restarts. 5. When leadership is lost, stop issuing cleanup DELETE requests. 6. Treat a Pod with `metadata.deletionTimestamp`, or an observed DELETE event, as already satisfying cleanup and do not enqueue it again. 7. Log an accepted DELETE request as such, treat an empty/not-found result as an idempotent no-op, and reserve warning/error logs for real exceptions. For a single-server deployment without HA coordination, the current local cleanup behavior can remain the fallback. Keeping informers only on the cleanup leader is not proposed here. Non-leader servers still submit and monitor their own engines and batches, while `getApplicationInfoByTag` currently reads the local informer-populated application store. Removing those informers would require a separate distributed state-propagation mechanism or repeated Kubernetes GET polling, which is a broader architectural change and may increase API-server traffic. ### Alternatives considered - **The submitting server deletes its own Driver Pods:** this naturally distributes cleanup, but the submitting server may fail or restart before the Driver terminates. Batch reassignment and shared engines can also outlive or move away from the original submitter. A fallback leader would still be required to prevent leaked Pods. - **A Kubernetes Lease-based leader:** this adds Lease RBAC requirements and continuous Lease traffic to the same Kubernetes API server whose request load we want to reduce. - **A distributed lock for every Pod deletion:** all servers would still contend for every Pod, and contenders may execute sequential DELETE requests after the lock is released unless additional completion state is persisted. ### Suggested tests - With two logical Kyuubi servers for the same context and namespace, only the elected leader issues a cleanup DELETE. - Different context/namespace pairs can have independent cleanup leaders. - After leader loss, the new leader reconciles and deletes an existing terminated Pod. - A Pod with `deletionTimestamp` or an observed DELETE event is not enqueued again. - An already-absent Pod does not produce a failure warning, while RBAC, throttling, network, and API-server failures preserve their exception details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
