[
https://issues.apache.org/jira/browse/SPARK-59722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125583#comment-18125583
]
Serge Rielau commented on SPARK-59722:
--------------------------------------
Thsi is proving to be very complex. I'm considering narrowing teh support for
map keys and opeing another Jira for this.
> Enforce CHAR/VARCHAR semantics for JSON map keys in schema-driven parsing
> -------------------------------------------------------------------------
>
> Key: SPARK-59722
> URL: https://issues.apache.org/jira/browse/SPARK-59722
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Serge Rielau
> Priority: Major
> Labels: pull-request-available
>
> {{from_json}} / JSON datasource maps with a first-class {{MAP<CHAR(n), ...>}}
> or {{MAP<VARCHAR(n), ...>}} schema still treat keys as unbounded STRING.
> Distinct serialized names can therefore collide after CHAR padding or VARCHAR
> trailing-space trimming ({{{}"a"{}}} vs {{"a "}} as {{{}CHAR(2){}}}),
> producing duplicate physical keys or silent last-wins instead of
> {{{}spark.sql.mapKeyDedupPolicy{}}}.
> SPARK-59274 covers scalar CHAR/VARCHAR assignment in JSON/CSV/XML,
> {{{}schema_of_*{}}}, and XML map keys. JSON map-key normalization is the
> remaining CHAR/VARCHAR surface.
> In scope
> * Apply assignment semantics to JSON object names when the declared map key
> type is CHAR/VARCHAR (pad CHAR; overflow -> {{{}EXCEED_LIMIT_LENGTH{}}}).
> * Honor {{mapKeyDedupPolicy}} for collisions created by that normalization;
> keep exact repeated names as historical last-wins.
> * Keep ordinary {{MAP<STRING, ...>}} JSON construction unchanged (including
> duplicate-entry cardinality).
> * Drain a failed nested JSON map value to its owning boundary before
> recovery continues (required so CHAR/VARCHAR key handling does not leak inner
> fields as outer keys).
> Out of scope
> * Ordinary STRING maps, including non-binary collated keys
> ({{{}UTF8_LCASE{}}} {{a}} vs {{{}A{}}}). That is pre-existing
> parser/collation behavior, not CHAR/VARCHAR length semantics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
