andygrove opened a new pull request, #7336: URL: https://github.com/apache/datafusion-comet/pull/7336
## Which issue does this PR close? N/A. There is no tracking issue. ## Rationale for this change An application has to set `spark.shuffle.manager` to `CometShuffleManager` itself. If it does not, Comet disables itself and logs a warning, so the plugin alone gives a user nothing. On Spark 3.4 and 3.5 the plugin cannot fix this, because `SparkEnv.create` builds the shuffle manager before any plugin runs. From Spark 4.0 (SPARK-45762) `SparkContext` creates it right after it initializes the plugins (`SparkContext.scala`: `PluginContainer(...)`, then `_env.initializeShuffleManager()`), from the same `SparkConf`. Executors take their conf from the driver's `sc.conf.getAll` when they register, which happens after the plugin has run, and their own `initializeShuffleManager()` also follows their plugin initialization. So the driver plugin can choose the manager for the whole application. ## What changes are included in this PR? - `CometDriverPlugin.init` calls a new `maybeSetShuffleManager`, which sets `spark.shuffle.manager` to `CometShuffleManager` on Spark 4.0 and later. It does so only when `spark.comet.enabled` and Comet shuffle are on and the application did not set `spark.shuffle.manager`. An explicit value is always kept, including `sort` and the Celeborn manager, and Comet then disables itself as before. It runs before the cache serializer check, which only installs Comet's serializer with one of Comet's shuffle managers. - Spark 3.4 and 3.5 are unchanged: the application still sets the manager itself. - The warning that Comet is disabled for lack of the manager now says the plugin sets it on 4.0 and later. The `spark.comet.shuffle.enabled` doc, the shuffle tuning guide and the FAQ describe the new behavior. ## How are these changes tested? - `CometDriverPluginShuffleManagerSuite` tests the decision: the version gate, a user-set value being kept (`sort`, `tungsten-sort`, Celeborn, a custom class), Comet or Comet shuffle disabled, and the deprecated `spark.comet.exec.shuffle.enabled` key. - `CometPluginsDefaultShuffleManagerSuite` starts a session with the plugin and no shuffle manager. On 4.0 and later it checks that `SparkEnv` holds `CometShuffleManager` and the plan has Comet shuffle. On 3.5 it checks that Comet stays disabled. - I ran both suites on the 3.5, 4.0 and 4.1 profiles, along with `CometPluginsSparkShuffleManagerSuite` and `CometPluginsUnifiedModeSuite` on 4.1. Executors only run in the local-mode tests, so the executor side rests on reading the Spark 4.0.0 source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
