Doris-Breakwater commented on issue #68080: URL: https://github.com/apache/doris/issues/68080#issuecomment-5696187735
Breakwater-GitHub-Analysis-Slot: slot_741ad7e71767 ## Initial assessment This is an actionable JDBC resource-lifecycle bug. The leak path is verified from the Doris code even though the exact production trigger sequence is not yet fully correlated. The issue is currently open, unassigned, and has no labels, milestone, or linked PR. ### Verified facts - In Doris 2.1.11 (and the current `branch-2.1` tip), [`BaseJdbcExecutor.init()` assigns `conn` from Hikari and then calls `initializeStatement()`](https://github.com/apache/doris/blob/a3052de2e5cbc397d67b9cd9d5cd9701aaa6426c/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/BaseJdbcExecutor.java#L398-L415). Its exception handlers rethrow without closing partially initialized resources. - `init()` runs inside the Java constructor. If it throws, JNI construction fails before native code sets `_is_open = true`; [`JdbcConnector::close()` returns immediately when `_is_open` is false](https://github.com/apache/doris/blob/a3052de2e5cbc397d67b9cd9d5cd9701aaa6426c/be/src/vec/exec/vjdbc_connector.cpp#L68-L73). Therefore the ordinary `BaseJdbcExecutor.close()` cleanup cannot be relied on for this path. - A failure in `setAutoCommit`, `prepareStatement`, or `setFetchSize` after `getConnection()` succeeds can consequently retain the Hikari proxy/`PoolEntry` in `IN_USE`. This is independent of why the borrowed connection is unusable. - The same ordering and lack of failure cleanup is present at the current tips of `branch-2.1`, `branch-3.0`, `branch-3.1`, `branch-4.0`, and `branch-4.1`. - One correction to the report: at current master commit `d9f837316734a0905064c02ff99d1aaab14e1786`, `BaseJdbcExecutor` was removed by [#66729](https://github.com/apache/doris/pull/66729). However, the equivalent problem remains in the replacement path: [`JdbcJniScanner.openInternal()` borrows a connection and performs statement/query/result initialization without cleanup in its catch block](https://github.com/apache/doris/blob/d9f837316734a0905064c02ff99d1aaab14e1786/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java#L163-L218), while native [`JniReader::close()` skips Java cleanup when `open()` failed before `_scanner_opened` became true](https://github.com/apache/doris/blob/d9f837316734a0905064c02ff99d1aaab14e1786/be/src/format/jni/jni_reader.cpp#L197-L204). - Current master has analogous partial-open gaps in `JdbcConnectionTester.openInternal()` and `JdbcJniWriter.openInternal()`, so the fix should audit all three entry points rather than only the removed executor class. - HikariCP 4.0.3 and 6.0.0 both contain the 500 ms `aliveBypassWindowMs` optimization. The reported timings and the successful `aliveBypassWindowMs=0` mitigation make the proposed dead-connection re-borrow trigger plausible. They are not required to establish the Doris ownership bug. ### Not yet proven / information still needed The supplied observations do not yet connect one connection ID through all of these events: SQL Server abort, Hikari recycle, sub-500 ms re-borrow, the exact executor-initialization exception, and absence of proxy close/recycle. To confirm that specific trigger and rule out a second leak path, please provide: 1. The exact Doris BE build Git SHA, JDK version, and Microsoft JDBC driver (`mssql-jdbc`) version. 2. The complete BE Java exception/stack trace for the first initialization failure after a fast re-borrow, including timestamps and pool name. The key question is which operation failed (`setAutoCommit`, `prepareStatement`, `setFetchSize`, or another initialization step). 3. A correlated trace for one connection/proxy identity showing borrow -> initialization exception -> no proxy `close()`/Hikari recycle. The physical SQL Server connection ID alone is insufficient to prove the final ownership transition. 4. Approximate reproduction rate and concurrency (for example, failures per N queries at M concurrent scans). A Doris query profile is not necessary for this lifecycle bug; the exception and lifecycle correlation above are more useful. ### Recommended next steps 1. Treat this as a valid JDBC Catalog bug and add the repository's JDBC/external-catalog bug labels if available. Prior reports [#33380](https://github.com/apache/doris/issues/33380) and [discussion #35780](https://github.com/apache/doris/discussions/35780) describe similar pool-exhaustion symptoms, but they do not contain enough evidence to call this issue a duplicate. 2. Add a deterministic failure-injection unit test: let `getConnection()` succeed, make each subsequent initialization stage throw, and assert that every created `ResultSet`/`Statement` is closed and the Hikari proxy is always closed/returned. Include the case where the physical delegate is already closed. 3. On release branches, make `BaseJdbcExecutor.init()` exception-safe by cleaning up partially created resources in reverse order before rethrowing the original exception. Cleanup failures should be logged or suppressed without replacing the initialization failure. 4. On master, apply the same guarantee to `JdbcJniScanner`, `JdbcConnectionTester`, and `JdbcJniWriter` partial-open paths. Also consider whether the shared native/JNI lifecycle should call idempotent Java cleanup after a failed `open()`, but the JDBC methods should remain exception-safe at the ownership boundary. 5. Backport the focused lifecycle fix to affected supported release branches. Keep `-Dcom.zaxxer.hikari.aliveBypassWindowMs=0` documented only as a temporary mitigation; it reduces this trigger but does not repair the leak for other post-borrow failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
