suxiaogang223 opened a new pull request, #70793: URL: https://github.com/apache/doris/pull/70793
### Background The Iceberg connector uses the SDK's shared worker pool for parallel manifest planning and manifest writes. All Iceberg catalogs on an FE, including Iceberg siblings of Hive catalogs, use the same plugin and pool. The SDK sizes this pool from a JVM property or environment variable, defaulting to max(2, available processors). ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Enhancement Problem Summary: Operators need a deployment-level way to set the shared Iceberg worker budget through the connector configuration. Master already shares the SDK pool across catalogs, but iceberg.conf does not expose its size. Add a startup-only setting and apply it before the first connector can initialize the SDK pool, covering both scan planning and SDK-internal manifest writes. ### Details - **Before and after:** Previously only the SDK JVM property/environment setting controlled the shared pool. Operators can now configure its total worker count in iceberg.conf; no per-catalog executor is introduced. - **Configs and defaults:** Add optional `worker_thread_pool_size` to the Iceberg plugin's `iceberg.conf`. It must be a positive integer and requires an FE restart on each node. When unset or blank, preserve `iceberg.worker.num-threads` / `ICEBERG_WORKER_NUM_THREADS` and the SDK default of max(2, available processors). An explicit plugin setting takes precedence over those SDK settings. The SDK queue and separate engine metadata, deletion, authentication-refresh and cancellation-cleanup pools are unchanged. - **Impact on existing users:** Untouched deployments retain their existing worker count. Explicit settings apply across standalone Iceberg catalogs and Hive catalog Iceberg siblings. Invalid values fail connector initialization with the setting name. An already initialized SDK pool with a different size is rejected rather than silently ignoring the configured limit. - **Behavior change:** An explicit plugin setting changes the shared SDK worker count for manifest planning and manifest writes. SDK ownership, catalog authentication, worker classloader pinning and scan cancellation/cleanup are preserved. ### Testing Local checks passed: - `cd fe && mvn -B -pl :fe-connector-iceberg checkstyle:check -Dmaven.build.cache.enabled=false` — zero Checkstyle violations, including test sources. - `bash build-support/check-fe-connector-imports.sh` - `bash build-support/check-fe-core-metadata-funnel.sh` - `git diff --cached --check` Added JUnit coverage for positive/invalid/blank configuration, concurrent initialization across catalog contexts, SDK setting precedence, shared-pool survival after scan cancellation/close, worker classloader pinning and premature SDK initialization. These tests have not been executed locally. Compilation and unit/regression tests were not run; local validation was limited to style and static checks. ### Release note Iceberg's shared FE worker pool can be sized with the optional positive `worker_thread_pool_size` setting in the plugin's `iceberg.conf`. The setting applies to all Iceberg catalogs on that FE and requires an FE restart. Leaving it unset preserves the existing SDK configuration and defaults. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No additional documentation; usage, defaults, precedence and restart requirements are documented in the shipped iceberg.conf.template. - Pre-submission status: Partially verified: local style/static checks passed; compilation and unit/regression tests have not been run. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, tests and this description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
