nsivabalan opened a new pull request, #20217: URL: https://github.com/apache/hudi/pull/20217
### Describe the issue this Pull Request addresses Once the S3 client's HTTP connection pool is shut down, it belongs to the JVM-wide cached file system: every later write task on that executor fails with `IllegalStateException: Connection pool shut down`, while the executor keeps heartbeating and keeps receiving tasks. The job then burns through task retries on a poisoned executor instead of getting a fresh one. We have seen this from several different S3 calls inside the merge handle (`getObjectMetadata` on open, `abortMultipartUpload` on close, `getFileStatus` for the file size) and from the fallback merge handle's constructor. All of them surface in `BaseSparkCommitActionExecutor.handleUpsertPartition`, with the pool error either the direct cause, nested several levels down under an `InvocationTargetException`, or attached as a suppressed exception from a failing `close()`. ### Summary and Changelog Adds an opt-in safeguard: when `hoodie.write.halt.executor.on.connection.pool.shutdown` is enabled (default **false**), an upsert write task that fails with the connection-pool-shutdown error halts the executor JVM (exit code 57) so Spark replaces the executor. - `HoodieWriteConfig`: new advanced config `hoodie.write.halt.executor.on.connection.pool.shutdown` + `shouldHaltExecutorOnConnectionPoolShutdown()`. - `ConnectionPoolShutdownExecutorHalter` (new, `hudi-spark-client`): walks the failure's cause chain and suppressed exceptions (cycle-safe, null-safe) and calls `Runtime.getRuntime().halt(57)` on a match. - `BaseSparkCommitActionExecutor.handleUpsertPartition`: in the existing `catch (Throwable)`, after the error log and before the rethrow, invoke the halter when the config is on. Design notes: - `Runtime.halt` rather than `System.exit`: shutdown hooks (e.g. the Hadoop `FileSystem` cache close) can block on the same dead client. - Exit code 57 is not used by Spark itself (0–3, 11, 50–56, 127), so the executor's exit reason identifies this case. - Detection matches the exact exception type and message raised by Apache HttpClient, so it is best-effort: if the message is ever reworded, this simply stops halting and behavior reverts to today's; an unrelated failure never halts the JVM. ### Impact New opt-in config only; no behavior change by default. When enabled: covers tasks going through `handleUpsertPartition` (plain inserts too, since `handleInsertPartition` delegates to it). Not covered: `SparkInsertOverwriteCommitActionExecutor`, which overrides `handleInsertPartition` without a catch block, and stages other than the write. Halting also kills the other tasks running on that executor; they are retried, and the executor's shuffle output is recomputed as with any executor loss. ### Risk Level low — disabled by default, and the match is narrow (exact exception type + message). ### Documentation Update New config `hoodie.write.halt.executor.on.connection.pool.shutdown` with its description in `HoodieWriteConfig`; it will be picked up by the generated config docs. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable `TestConnectionPoolShutdownExecutorHalter` covers: the pool error thrown directly, wrapped by the upsert exception, nested under the fallback handle constructor, and suppressed by a failing close (all halt); another `IllegalStateException`, the same message on another type, an unrelated failure, an unrelated suppressed exception, cyclic chains (direct and via suppressed), and a null failure (none halt); and the config defaulting to false. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
