andygrove opened a new pull request, #5181:
URL: https://github.com/apache/datafusion-comet/pull/5181
## Which issue does this PR close?
N/A — routine patch-version bump within the already-supported Spark 3.5 line.
## Rationale for this change
Spark 3.5 support should track the latest Spark 3.5 patch release. Apache
Spark
3.5.9 has been released (tag `v3.5.9`), so this moves Comet's Spark 3.5
build,
CI, and documentation from 3.5.8 to 3.5.9.
## What changes are included in this PR?
- Bumps `<spark.version>` in the `spark-3.5` Maven profile to 3.5.9. Spark
3.5.9
makes no transitive dependency changes relative to 3.5.8 (`scala.version`,
`parquet.version`, `slf4j.version`, `jetty.version`, `hadoop.version` and
`arrow.version` are all unchanged), so no other profile properties move.
- Renames `dev/diffs/3.5.8.diff` to `dev/diffs/3.5.9.diff` and regenerates it
against the `v3.5.9` tag. Three hunks needed manual reapplication:
- `pom.xml` — context drift only (`ivy.version` bumped upstream).
- `WriteDistributionAndOrderingSuite.scala` — context drift only (upstream
added `TransformExpression` to a neighbouring import).
- `KeyGroupedPartitioningSuite.scala` — context drift plus one real change:
3.5.9 adds a new `collectAllShuffles` helper matching on
`ShuffleExchangeExec`. Under Comet the shuffle is a
`CometShuffleExchangeExec`, so the new helper's
`assert(allShuffles.nonEmpty)`
call sites would fail. It now matches `ShuffleExchangeLike`, consistent
with
how the existing `collectShuffles` helper is already patched.
- Points the Spark SQL, Spark SQL writer, and Iceberg 1.9/1.10 CI jobs at
3.5.9,
and updates `dev/ci/compute-changes.py`'s `spark_3_5` path filter to the
new
diff filename.
- Updates the user guide's supported-version table, the Spark version
compatibility page, the Iceberg contributor guide, the
`setup-spark-builder`
action / reusable-workflow input descriptions, and the `pip install
pyspark==`
examples in the PyArrow UDF test scripts.
## How are these changes tested?
- `git apply --check dev/diffs/3.5.9.diff` against a pristine `v3.5.9`
checkout
passes with no rejects or fuzz.
- Verified the regenerated diff touches exactly the same 73 files as the
3.5.8
diff, and that its only content differences from 3.5.8 are the context
lines
above plus the `collectAllShuffles` change.
- `npx prettier --check` on the modified markdown.
- Full verification is the Spark SQL CI job on this PR, which clones
`v3.5.9`,
applies the new diff, and runs the SQL test suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
