andygrove opened a new pull request, #4986:
URL: https://github.com/apache/datafusion-comet/pull/4986
## Summary
Part of #4978 (Category 2 + Category 4).
Shuffle-related configs live under four disjoint prefixes today:
`spark.comet.exec.shuffle.*`, `spark.comet.columnar.shuffle.*`,
`spark.comet.native.shuffle.*`, and `spark.comet.shuffle.*`. This PR
consolidates all shuffle configs under `spark.comet.shuffle.*` with
`.columnar.` and `.native.` sub-namespaces for mode-specific settings,
and fixes the dots-in-segment naming for the three columnar keys.
Every renamed key is registered via `.withAlternative(...)`, so
existing user configurations keep working (a one-time deprecation
warning is logged per old key). No behavior change.
## Rename table
**Master / shared (both modes):**
| Old key | New key |
| --- | --- |
| `spark.comet.exec.shuffle.enabled` | `spark.comet.shuffle.enabled` |
| `spark.comet.exec.shuffle.mode` | `spark.comet.shuffle.mode` |
| `spark.comet.exec.shuffle.directRead.enabled` |
`spark.comet.shuffle.directRead.enabled` |
| `spark.comet.exec.shuffle.convertFromSparkPlan.enabled` |
`spark.comet.shuffle.convertFromSparkPlan.enabled` |
| `spark.comet.exec.shuffle.revertRedundantColumnar.enabled` |
`spark.comet.shuffle.revertRedundantColumnar.enabled` |
| `spark.comet.exec.shuffle.compression.codec` |
`spark.comet.shuffle.compression.codec` |
| `spark.comet.exec.shuffle.compression.zstd.level` |
`spark.comet.shuffle.compression.zstd.level` |
**Native-shuffle-only** (moved from `spark.comet.native.shuffle.*` and one
leaf from `spark.comet.exec.shuffle.*`):
| Old key | New key |
| --- | --- |
| `spark.comet.exec.shuffle.writeBufferSize` |
`spark.comet.shuffle.native.writeBufferSize` |
| `spark.comet.native.shuffle.partitioning.hash.enabled` |
`spark.comet.shuffle.native.partitioning.hash.enabled` |
| `spark.comet.native.shuffle.partitioning.range.enabled` |
`spark.comet.shuffle.native.partitioning.range.enabled` |
| `spark.comet.native.shuffle.partitioning.roundrobin.enabled` |
`spark.comet.shuffle.native.partitioning.roundrobin.enabled` |
| `spark.comet.native.shuffle.partitioning.roundrobin.maxHashColumns` |
`spark.comet.shuffle.native.partitioning.roundrobin.maxHashColumns` |
**Columnar / JVM-shuffle-only** (moved from
`spark.comet.columnar.shuffle.*`; also fixes Category 2 dots-in-segment):
| Old key | New key |
| --- | --- |
| `spark.comet.columnar.shuffle.spill.threshold` |
`spark.comet.shuffle.columnar.spillThreshold` |
| `spark.comet.columnar.shuffle.memory.factor` |
`spark.comet.shuffle.columnar.memoryFactor` |
| `spark.comet.columnar.shuffle.batch.size` |
`spark.comet.shuffle.columnar.batchSize` |
| `spark.comet.shuffle.preferDictionary.ratio` |
`spark.comet.shuffle.columnar.preferDictionary.ratio` |
`preferDictionary.ratio` and `writeBufferSize` were previously at
mode-agnostic namespaces but their doc/call sites confirm they are
JVM-only and native-only respectively; the rename puts them in their
honest place.
## Not in this PR
- `spark.comet.columnar.shuffle.max.writers.per.executor` isn't
renamed here — the val `COMET_COLUMNAR_SHUFFLE_MAX_WRITERS_PER_EXECUTOR`
is introduced by #4980 (async columnar shuffle removal), which hasn't
landed on main yet. A tiny follow-up PR can add its rename once #4980
merges.
- The `spark.comet.columnar.shuffle.async.*` keys are being removed by
#4980 — left alone here.
## Test plan
- [x] `./mvnw -Pspark-3.5 test -pl spark
-Dsuites="org.apache.comet.CometConfSuite"` — 10 tests pass (including 3 new
alias tests for the renames)
- [x] `./mvnw -Pspark-3.5 spotless:check` — clean
- [x] Full `compile test-compile` under Spark 3.5 — clean
- [ ] CI runs the broader test matrix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
