comphead opened a new pull request, #7302: URL: https://github.com/apache/datafusion-comet/pull/7302
## Which issue does this PR close? Closes #7301. ## Rationale for this change A job launched from a query's RDD (`Dataset.rdd` with AQE off, `QueryExecution.toRdd` with AQE on or off) runs its map stages outside a SQL execution. On an executor, `SQLConf.get` then reads task local properties that Spark has not filled, and `SQLExecutionRDD` (SPARK-28939) re-applies the session's configs only in the final stage. Comet reads its settings on the executor when it builds a native plan, so those map stages ran with every `spark.comet.*` setting at its default. That includes the memory pool, the batch size, the shuffle codec and the configs passed to native. There was no warning. ## What changes are included in this PR? - `CapturedSQLConf` captures the session's configs on the driver. In a task outside a SQL execution, `withConf` makes them what `SQLConf.get` returns on the task thread, using `SQLConf.withExistingConf` the same way `SQLExecutionRDD` does. Tasks inside a SQL execution are unchanged. - It applies where Comet builds native plans on executors: `CometExecRDD.compute`, `CometNativeShuffleInputRDD.compute`, `CometNativeShuffleWriter.write` (through a new `NativeShuffleSpec.sqlConf`), and the native limit and top-K closures in `CometExecUtils` and `CometTakeOrderedAndProjectExec`. Not covered here: the JVM columnar shuffle writers also read settings such as the codec and spill threshold on the executor, and still see defaults in these jobs. Native writes run inside a SQL execution, so they are not affected. ## How are these changes tested? A new `CometNativeShuffleSuite` test sets `spark.comet.batchSize=10`, runs a 1000-row repartition through `toRdd` with AQE off, and checks that the shuffle writer received 100 batches. The same scenario in `spark-shell` (Spark 4.1.1) gives 1 batch before this change and 100 after. The probe from #7301 run against this branch resolves the submitted `greedy_unified` pool and 0.5 fraction in the map stage in all five scenarios, where three of them resolved `fair_unified` and 1.0 before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
