ruanwenjun opened a new issue, #7708: URL: https://github.com/apache/kyuubi/issues/7708
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched the existing open and closed issues and found no similar issue. ### Describe the bug A persistent JDBC metadata update failure can remain in the asynchronous retry queue indefinitely and block batch session cleanup. The relevant sequence is: 1. `JDBCMetadataStore.updateMetadata` throws when `PreparedStatement.executeUpdate()` returns `0`. 2. With `kyuubi.metadata.request.async.retry.enabled=true`, `MetadataManager.updateMetadata` enqueues the failed update. 3. The retry worker removes the queue head only after a successful update. A deterministic failure therefore remains at the head and is retried indefinitely. 4. `KyuubiBatchSession.close` waits without a timeout until the metadata retry queue is empty. 5. The session timeout checker is a single-thread scheduled executor and calls `closeSession` synchronously. One affected batch session can therefore prevent later sessions from being checked and removed. Old sessions may continue to appear in `/api/v1/sessions` even while their metadata rows are independently removed by the metadata cleaner. In the observed sequence, a batch session was closed while its application monitor was still running. The monitor later observed the driver as `FAILED`, updated the application information, and the asynchronous operation's `finally` block attempted another terminal metadata update. This explains repeated terminal updates, but it has **not** been established that repetition alone caused Connector/J to return `0`. The exact reason for the first zero update count is still under investigation. This report is about the deterministic failure amplification after any persistent zero-count update, not an assumption about its original cause. #### Reproduction outline 1. Configure the MySQL JDBC metadata store with asynchronous retry enabled. 2. Submit a batch and make a terminal metadata update return `0`, for example by deleting the row before the update or by using a test metadata store that produces a deterministic zero-count update. 3. Close the corresponding batch session. 4. Observe that the same update remains at the retry queue head. 5. Observe that `KyuubiBatchSession.close` does not return and that the single session timeout checker cannot process later sessions. #### Expected behavior - Metadata persistence failures must not block session lifecycle cleanup indefinitely. - A zero-count update should be reconciled using its postcondition instead of being treated as unconditional success or unconditional transient failure: - if the row exists and all columns included in the update already match the intended values, the operation is idempotently complete; - if the row is missing or its values do not match, retain a classified failure without silently recreating deleted metadata; - preserve ordering when an earlier INSERT for the same identifier is pending. - Persistent failures should use observable and bounded/rate-limited handling, and normal session close should not wait indefinitely for the asynchronous retry queue. ### Affects Version(s) Observed on a deployment based on 1.9.1. The relevant code paths are also present on master at `804c00239815d3b7eece5c4c8bb8f6263063beff`. ### Kyuubi Server Log Output ```logtalk INFO ... KyuubiBatchSession: Mark session SessionHandle [<batch-id>] closed INFO ... BatchJobSubmission: Batch report for <batch-id>, Some(ApplicationInfo(...,FAILED,...)) ERROR ... MetadataManager: Error updating metadata for session <batch-id> org.apache.kyuubi.KyuubiException: Error updating metadata for <batch-id> by SQL: UPDATE metadata SET ... WHERE identifier = ? INFO ... MetadataManager: Retrying metadata requests for <batch-id> ERROR ... MetadataManager: Error retrying metadata requests for <batch-id> INFO ... KyuubiBatchSession: There are still remaining metadata store requests for batch[<batch-id>] ``` ### Kyuubi Engine Log Output Not required to reproduce the metadata retry and session-close behavior. ### Kyuubi Server Configurations ```yaml kyuubi.metadata.store.jdbc.database.type: MYSQL kyuubi.metadata.request.async.retry.enabled: true kyuubi.metadata.request.retry.interval: 5s ``` The deployment is intended to use MySQL Connector/J 5.x and the JDBC URL does not explicitly set `useAffectedRows=true`; the effective runtime driver and connection properties are still being verified. ### Additional context Relevant current-master code: - [`JDBCMetadataStore.updateMetadata`](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala#L393-L401) - [metadata asynchronous retry loop](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala#L322-L380) - [`KyuubiBatchSession` waiting for retry completion](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiBatchSession.scala#L127-L136) ### Are you willing to submit PR? - [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
