sunchao opened a new pull request, #6495:
URL: https://github.com/apache/datafusion-comet/pull/6495
## Which issue does this PR close?
Closes #6233. Follow-up to #5302.
## Rationale for this change
`CometCollationSuite` has separate Spark 4.0 and 4.1 copies and is absent
from Spark 4.2. This leaves 4.2 without its collation fallback and datetime
coverage, while the duplicated suites can drift.
## What changes are included in this PR?
Move the common tests and both sort-merge join tests to `spark-4.x`, and
remove the duplicate 4.1 suite. Keep the three hash-join tests in a Spark
4.0-only `CometHashJoinCollationSuite`, registered in both CI workflows. Spark
4.1+ normalizes collated hash-join keys to binary keys before the converter
sees them, so those guard tests remain specific to 4.0.
The existing test bodies and assertions are preserved. Spark 4.0 retains all
28 tests (25 shared + 3 hash-join tests); Spark 4.1 gains the two sort-merge
join tests; Spark 4.2 gains all 25 shared tests. Production behavior is
unchanged.
## How are these changes tested?
Local checks at `260ff8a5a`, using JDK 21 and a debug native library built
from the current base:
- `make core`: passed.
- Spark 4.0.4: all 28 tests passed (shared suite plus
`CometHashJoinCollationSuite`).
- Spark 4.1.3: all 25 shared tests passed.
- Spark 4.2.0: all 25 shared tests passed.
- All three profile runs completed the full Maven reactor successfully,
including Spotless and Scalastyle. No tests were ignored or canceled.
- `python3 dev/ci/check-suites.py` and `git diff --check`: passed.
The Spark profile runs used separate checkouts to avoid mixing compiled
classes. Example commands, run from the repository root:
```sh
./mvnw -Pspark-4.0 test -Dtest=none \
-Dsuites=org.apache.spark.sql.CometCollationSuite,org.apache.spark.sql.CometHashJoinCollationSuite
./mvnw -Pspark-4.1 test -Dtest=none
-Dsuites=org.apache.spark.sql.CometCollationSuite
./mvnw -Pspark-4.2 test -Dtest=none
-Dsuites=org.apache.spark.sql.CometCollationSuite
```
CI results are pending; `run-all-spark-profiles` requests coverage beyond
the default 4.1 profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
