andygrove opened a new pull request, #7205: URL: https://github.com/apache/datafusion-comet/pull/7205
## Which issue does this PR close? Closes #7118. Backport of #7123 (squash commit 2b1c6a3938) to `branch-1.2`. ## Rationale for this change #6786 is already on `branch-1.2`. It made Comet convert a Text scan on a broadcast join's build side to Arrow by default, where 1.1.0 left it on Spark. Arrow strings must be valid UTF-8, so Comet replaces each ill-formed sequence with `U+FFFD`, and `hex`, `length`, hashing, equality and ordering over a Latin-1 Text file can then differ from Spark. Without this fix, 1.2.0 would return different results from 1.1.0 for the same query. See #7123 for the full rationale. ## What changes are included in this PR? A clean cherry-pick of #7123 with `-x`. The changed lines are identical to the source commit, and no file was adapted: - New `spark.comet.convert.text.enabled` (default `false`), so a Text scan is converted only by its own config, like CSV, JSON and Parquet. `spark.comet.convert.broadcastBuildSide.enabled` stays on, but a Text build side is bridged only when the Text config is set. - Compatibility guide, `datasources.md` and migration guide updates. - The #6008 build-side tests move from `CometExecSuite` to `CometBroadcastBuildSideSuite`, and run against an ORC build side. `docs/source/user-guide/latest/configs.md` on `branch-1.2` still holds the `CONFIG_TABLE` markers rather than generated tables, so there is no row to update. Backports to other branches: none needed. #6786 is not on `branch-1.0` or `branch-1.1`, so neither has the behavior this fixes. ## How are these changes tested? On this branch, with Spark 4.1 (the default profile): - `CometBroadcastBuildSideSuite` and `CometExecSuite` ran 194 tests: 193 passed, one was canceled (a Spark 3.4-only test) and one failed. The failure is "native scan loads the next file's metadata while it reads the current file" (`0 did not equal 3`). It exercises #7094, which merged after the `libcomet` I used was built, and this PR does not touch it. I did not rerun it with a freshly built library. - `./mvnw test-compile -Pspark-3.5,delta -Pstrict-warnings` and the Scalafix `CHECK` run both pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
