andygrove opened a new pull request, #5183:
URL: https://github.com/apache/datafusion-comet/pull/5183
## Which issue does this PR close?
N/A — routine patch-version bump within the already-supported Spark 4.1 line.
## Rationale for this change
Spark 4.1 support should track the latest Spark 4.1 patch release. Apache
Spark
4.1.3 has been released (tag `v4.1.3`), so this moves Comet's Spark 4.1
build,
CI, Docker image, and documentation from 4.1.2 to 4.1.3. Spark 4.1 is also
Comet's default profile, so this bumps the top-level `<spark.version>` too.
## What changes are included in this PR?
- Bumps the default `<spark.version>` and the `spark-4.1` profile's
`<spark.version>` to 4.1.3. Spark 4.1.3 makes no transitive dependency
changes
relative to 4.1.2 (`scala.version` 2.13.17, `parquet.version` 1.16.0,
`slf4j.version` 2.0.17, `jetty.version` 11.0.26, `hadoop.version` 3.4.2,
`arrow.version` 18.3.0 are all unchanged), so no other profile properties
move
and `spark/pom.xml`'s pinned Jetty 11.0.26 test deps still match — only the
comment naming the reference version is updated.
- Renames `dev/diffs/4.1.2.diff` to `dev/diffs/4.1.3.diff` and regenerates it
against the `v4.1.3` tag. Two hunks needed manual reapplication, both pure
import-line context drift with no behavioural change:
- `DataFrameWindowFunctionsSuite.scala` — upstream added a
`catalyst.trees.UnaryLike` import between the patched neighbours.
- `StreamRealTimeModeAllowlistSuite.scala` — upstream moved
`LowLatencyMemoryStream` from `execution.streaming` to
`execution.streaming.sources`.
- Points the Spark SQL, Spark SQL writer, Iceberg 1.11, and PyArrow UDF CI
jobs
at 4.1.3, and updates `dev/ci/compute-changes.py`'s `spark_4_1` path
filter to
the new diff filename.
- Bumps `kube/Dockerfile` to `apache/spark:4.1.3` and updates the Kubernetes,
macOS benchmarking, installation, compatibility, and Iceberg docs.
## How are these changes tested?
- `git apply --check dev/diffs/4.1.3.diff` against a pristine `v4.1.3`
checkout
passes with no rejects or fuzz.
- Verified the regenerated diff touches exactly the same 96 files as the
4.1.2
diff, and that its only content differences from 4.1.2 are the two context
lines above.
- `npx prettier --check` on the modified markdown.
- Full verification is the Spark SQL and Iceberg 1.11 CI jobs on this PR,
which
clone `v4.1.3`, apply the new diff, and run the test suites.
### One thing to confirm on review
Maven Central, PyPI, and Docker Hub are all unreachable from the environment
I
prepared this in, so I could not independently confirm that the
`org.apache.spark:*:4.1.3` artifacts, `pyspark==4.1.3`, and the
`apache/spark:4.1.3` image are all published yet — the `v4.1.3` tag
(identical to
`v4.1.3-rc1`, i.e. the vote passed on rc1) is the only evidence I could
gather.
CI will fail fast on the Maven and PyPI pins if either is not yet available.
The
`apache/spark` Docker image sometimes lags the release, and
`kube/Dockerfile` is
not exercised by CI, so that tag is worth an explicit check before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
