Doris-Breakwater commented on issue #67446: URL: https://github.com/apache/doris/issues/67446#issuecomment-5509581503
## Initial triage This is a valid, high-impact FE/Iceberg reliability symptom: once the executor reaches `Terminated`, recreating a catalog cannot recover writes and an FE restart is expected to be the only practical recovery. The issue currently has no labels; `kind/bug` plus the repository's Iceberg/FE ownership label (if available) would be appropriate. ### Verified from `ddbaaab1388` - Doris builds the FE connector against Iceberg **1.10.1**; `apache/iceberg-rest-fixture:1.10.0` is the server fixture version, not the version of `ThreadPools` running in the FE. - Iceberg's `WORKER_POOL` is a `static final` executor [created once by `ThreadPools`](https://github.com/apache/iceberg/blob/apache-iceberg-1.10.1/core/src/main/java/org/apache/iceberg/util/ThreadPools.java). More precisely, it is singleton-per-Iceberg-classloader; with the child-first connector plugin, it is shared by all Iceberg catalogs served by that plugin on this FE. - `ThreadPoolExecutor` cannot reach `Terminated` merely because tasks failed or workers exited. Shutdown must have been initiated (`shutdown`/`shutdownNow`, including the registered JVM shutdown hook). The exception therefore proves that the pool was already shut down, but it does **not** identify who initiated shutdown. - In Doris source at this commit, the only executable connector reference to `ThreadPools.getWorkerPool()` is the one-time TCCL primer in [`IcebergConnector.pinIcebergWorkerPoolToPluginClassLoader`](https://github.com/apache/doris/blob/ddbaaab13882dab3dfa51ffb8038218df869e360/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java#L1411-L1485). It submits primer tasks and never shuts the pool down. Once its `ICEBERG_WORKER_POOL_PINNED` flag is true, later catalog creation does not probe the pool again, which explains why a fresh catalog can be created while subsequent writes still fail. - The normal write teardown is also not an executor close: [`IcebergConnectorTransaction.close()`](https://github.com/apache/doris/blob/ddbaaab13882dab3dfa51ffb8038218df869e360/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java#L1400-L1412) only releases the tracked table/resource lease. - I audited the explicit cleanup added by [#66914](https://github.com/apache/doris/pull/66914), because it is the strongest recent suspect. Doris may close a REST catalog generation or an owned table `FileIO`, but Iceberg 1.10.1's corresponding close paths shut down their own REST auth/session executors and S3 clients. Those executors are created separately (`newScheduledPool` / `newExitingWorkerPool`) and are not `ThreadPools.getWorkerPool()`. I found no direct path from those closes to the failed singleton. Therefore catalog/FileIO cleanup is a useful bisect boundary, but is **not a verified root cause**. ### Current judgment The failure mode and blast radius are confirmed, but the initiating shutdown call is still unknown. There is not enough evidence to attribute it to `DROP CATALOG`, a failed write, cache retirement, or the REST fixture. A speculative replacement/recreation of the static pool would mask the shutdown owner and may introduce cross-catalog races; the shutdown caller should be captured first. ### Information needed 1. Please attach the complete stack trace for the **first** rejected write and the FE log covering roughly 10 minutes before it, including the last successful Iceberg operation, any cancelled/failed suite, `REFRESH CATALOG`, FE stop signal, or plugin rebuild/reload activity. Please also confirm the FE PID stayed unchanged throughout. 2. Please provide the exact suite invocation/order and the correct tracking PR/branch. The referenced `#67366` currently resolves to an unrelated Arrow Flight SQL issue. 3. On the next reproduction, record `ThreadPools.getWorkerPool().isShutdown()`, `isTerminated()`, and its identity after each top-level test/statement. This will locate the first operation that changes the lifecycle state. ### Recommended next steps - Highest-value experiment: instrument or attach a debugger to `ExecutorService.shutdown()` and `shutdownNow()` and capture a stack only when the target is the Iceberg connector's worker-pool delegate. That stack should identify the root cause directly; ordinary logs after rejection cannot reconstruct it. - Run the reported sequence on `ddbaaab1388`, then A/B at `25c2185405d^` (before #66914). The suites set `meta.cache.iceberg.table.ttl-second=0`, so the disabled-cache cleanup path is exercised frequently; a clean A/B result would either focus or eliminate that boundary without assuming it is causal. - After the pool becomes terminated, try one metadata-only command and one scan requiring multiple manifests. Iceberg also uses this pool for parallel planning, so reads may have a wider conditional impact even though writes are the deterministic reproducer. - Keep FE restart as the operational workaround. Catalog/database recreation cannot repair a final static executor in the same plugin classloader. Breakwater-GitHub-Analysis-Slot: slot_0b6313cb80ab -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
