andygrove opened a new pull request, #7337: URL: https://github.com/apache/datafusion-comet/pull/7337
## Which issue does this PR close? Backport of #7302 to `branch-1.2`. #7302 closes #7301 on `main`. ## Rationale for this change A job launched from a query's RDD (`Dataset.rdd` with AQE off, `QueryExecution.toRdd`) runs its map stages outside a SQL execution, so on an executor those stages ran with every `spark.comet.*` setting at its default (memory pool, batch size, the configs passed to native), with no warning (#7301, `bug`). `branch-1.2` has this bug, so it needs the fix before 1.2.0 ships. `branch-1.2` is the newest release branch, so no newer branch needs a backport. #7302 carries only the `backport-1.2` label, so no `branch-1.1` backport is part of this. ## What changes are included in this PR? `git cherry-pick -x 15305af627` of the squash commit of #7302. One conflict, in the import block of `CometNativeShuffleSuite.scala`: `branch-1.2` already lacks the `RangePartitioning` import that `main` still has, so the resolution keeps `branch-1.2`'s `HashPartitioning` import and takes #7302's added `CometCollectLimitExec` import. The added and removed lines are otherwise identical to the source commit. See #7302 for the full description. ## How are these changes tested? On this branch, with a `libcomet` built from the pick (the fix is JVM-only): - `./mvnw test-compile` passes on the default profile (Spark 4.1) and on `-Pspark-3.5`. - `CometNativeShuffleSuite`, `CometJoinSuite`, `CometTaskMetricsSuite`, `NativeUtilSuite` and `CometNativeShuffleInputRDDSuite` pass: 235 succeeded, 0 failed. That includes the five new `... outside a SQL execution uses the session's Comet configs` tests. I did not re-run the new tests without the fix on this branch (#7302 shows them failing before the change), and I did not run the Spark SQL test suites that #7302's CI ran. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
