andygrove commented on PR #4908: URL: https://github.com/apache/datafusion-comet/pull/4908#issuecomment-5096716098
Thanks for the thorough review, @mbutrovich! I merged main and pushed fixes for all seven comments in 266f251dc: - **Parallel I/O section** — dropped entirely; the four configs were removed in #4981. - **`maxTempDirectorySize` scope** — reworded in both `tuning.md` and the `CometConf` doc string to make clear the limit is per Spark task, so an executor running N concurrent tasks may use up to N× this value on shared disks. - **`### Parquet Native Scans`** — moved from under `## Shuffle` to the new `## Parquet Reader Tuning` section so all Parquet guidance is co-located. - **`Reducing Row/Columnar Conversion Overhead`** — promoted from h3-under-Shuffle to its own h2, since `transitionRevert.*` counts C2R transitions across the whole stage and isn't shuffle-specific. - **Shuffle compression** — rewrote the section to distinguish scopes: `spark.comet.exec.shuffle.compression.codec` governs both native and JVM columnar shuffle writers; `spark.shuffle.compress=false` only disables compression on the native path (JVM columnar always compresses). - **Iceberg wording** — fixed to say `spark.comet.scan.icebergNative.enabled` is enabled by default, and noted `dataFileConcurrencyLimit` defaults to 1. PTAL when you get a chance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
