andygrove opened a new issue, #27342: URL: https://github.com/apache/datafusion/issues/27342
### Is your feature request related to a problem or challenge? `datafusion-spark` is moving to [apache/datafusion-spark](https://github.com/apache/datafusion-spark) (#27103, #27337). Two settings in `datafusion-common` are read only by that crate: - [`SparkOptions`](https://github.com/apache/datafusion/blob/e7c51c7fb5ebd301f5eb239afbea159e2cfbcfb6/datafusion/common/src/config.rs#L2251-L2267) (`datafusion.spark.*`, from #21720). Its one option, `map_key_dedup_policy`, is read by the crate's `map_from_arrays`, `map_from_entries` and `str_to_map`. - [`datafusion.execution.enable_ansi_mode`](https://github.com/apache/datafusion/blob/e7c51c7fb5ebd301f5eb239afbea159e2cfbcfb6/datafusion/common/src/config.rs#L1166-L1187) (from #18635). Its docs say it is "relevant only for DataFusion Spark built-in functions". The crate's `abs`, `elt`, `mod`, `pmod`, `negative`, `round` and `spark_cast` read it. Core only has the `SessionConfig::with_enable_ansi_mode` setter. After the move, a new Spark option would need a DataFusion PR and then a DataFusion release before `datafusion-spark` could use it. #27260 and #27273 already add `spark.sql.legacy.*` flags to `SparkOptions`. Every DataFusion session also lists these settings in `SHOW ALL`, whether or not it uses the Spark crate. ### Describe the solution you'd like Move `SparkOptions` into `datafusion-spark` as a [`ConfigExtension`](https://docs.rs/datafusion-common/latest/datafusion_common/config/trait.ConfigExtension.html), the way `datafusion-cli` defines its [`aws.*` options](https://github.com/apache/datafusion/blob/e7c51c7fb5ebd301f5eb239afbea159e2cfbcfb6/datafusion-cli/src/object_storage.rs#L423-L424): 1. In apache/datafusion-spark, add the extension, register it in `SessionStateBuilderSpark::with_spark_features()`, and have the functions read it from `config_options.extensions`, falling back to the defaults when it isn't registered. This builds against both DataFusion 56 and `main`. 2. Then remove `SparkOptions` and `MapKeyDedupPolicy` from DataFusion 57, with an upgrade note in both repositories. The key has to change, for example from `datafusion.spark.map_key_dedup_policy` to `spark.map_key_dedup_policy`, because [`ConfigOptions::set`](https://github.com/apache/datafusion/blob/e7c51c7fb5ebd301f5eb239afbea159e2cfbcfb6/datafusion/common/src/config.rs#L2388-L2393) sends every `datafusion.` key to the built-in options. Removing the old setting rather than deprecating it means the old key returns an error instead of silently doing nothing. `enable_ansi_mode` should move the same way unless core will read it. In #23929, @comphead expects a few targeted core changes, such as overflow handling in window `RANGE` frames, safe casts and arithmetic. @alamb and I suggested scoping ANSI mode to `datafusion-spark`. If core does read it, it stays where it is. ### Describe alternatives you've considered Keep both settings in core. No keys change, but every new Spark option has to wait for a DataFusion release. ### Additional context Comet uses neither setting. Sail has its own `map_key_dedup_policy`. It does set `execution.enable_ansi_mode` on the `ConfigOptions` snapshots that it passes to `datafusion-spark`'s `SparkNegative` ([spark_negative.rs](https://github.com/lakehq/sail/blob/77568e78e0bbe81ddce7c46e2000edd70a550b41/crates/sail-function/src/scalar/math/spark_negative.rs#L9-L32)), so moving the ANSI flag would need a matching change there. Its comments explain why it fixes the flag at planning time: the session-level flag only reflects the driver's context. cc @comphead @shehabgamin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
