peterxcli opened a new issue, #5857:
URL: https://github.com/apache/datafusion-comet/issues/5857
### Describe the bug
With `spark.sql.mapKeyDedupPolicy=LAST_WIN`, `str_to_map` raises
`[DUPLICATED_MAP_KEY]` on an input string that repeats a key, while Spark
returns a map holding the last value for that key. Comet fails the query where
Spark succeeds.
The error also surfaces as a raw native failure rather than Spark's
`SparkRuntimeException`, because nothing translates it on the JVM side.
### Steps to reproduce
```sql
SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
SELECT str_to_map('a:1,b:2,a:3');
```
Spark returns `{a -> 3, b -> 2}`. Comet raises:
```
[DUPLICATED_MAP_KEY] Duplicate map key 'a' was found, please check the input
data.
To allow duplicate keys with last-value-wins semantics, set
`datafusion.spark.map_key_dedup_policy` to `LAST_WIN`.
```
The default `EXCEPTION` policy is unaffected: both engines raise on a
duplicate key there, so this only shows up once the policy is changed.
### Expected behavior
`str_to_map` follows `spark.sql.mapKeyDedupPolicy` the way Spark's
`ArrayBasedMapBuilder` does, keeping the last value for each duplicate key
under `LAST_WIN`.
### Additional context
The native kernel already implements this. `datafusion-spark`'s
`SparkStrToMap` reads `datafusion.spark.map_key_dedup_policy` (see
`datafusion_common::config::SparkOptions`), which takes the same `EXCEPTION`
and `LAST_WIN` values as the Spark config. Two things stop the Spark setting
from reaching it:
1. `spark.sql.mapKeyDedupPolicy` never crosses JNI.
`CometExecIterator.serializeCometSQLConfs` sends only keys under
`spark.comet.`, plus a short explicit list.
2. `create_scalar_function_expr` in `native/core/src/execution/planner.rs`
hands every `ScalarFunctionExpr` a fresh `ConfigOptions::default()`, so a
kernel reading a session option sees DataFusion's defaults rather than the
session's values.
`CometStrToMap.getSupportLevel` does not inspect the policy either, so the
expression stays `Compatible` and takes the native path under `LAST_WIN`
instead of routing through the JVM codegen dispatcher.
`spark/src/test/resources/sql-tests/expressions/map/str_to_map.sql` carries
a matching note:
```sql
-- TODO: Add LAST_WIN policy tests when spark.sql.mapKeyDedupPolicy config
is supported
```
#5854 fixes both points above while closing #4680, and adds
`str_to_map_dedup_policy.sql` covering `LAST_WIN`. This issue records the
divergence on its own, since it reaches a different expression from the one
#4680 and #5589 describe and would otherwise go untracked if that PR is split
up.
Related: #4680 (null keys in `map_from_arrays` / `map_from_entries`), #5589
(`map_from_arrays` fallback under `LAST_WIN`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
