[
https://issues.apache.org/jira/browse/SPARK-60103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60103:
-----------------------------------
Labels: pull-request-available (was: )
> Apply XML CHAR/VARCHAR assignment after STRING parse
> ----------------------------------------------------
>
> Key: SPARK-60103
> URL: https://issues.apache.org/jira/browse/SPARK-60103
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Serge Rielau
> Priority: Major
> Labels: pull-request-available
>
> Context
> SPARK-59274 already put CHAR/VARCHAR map keys in
> StaxXmlParser.convertConstrainedMap, plus attribute and rowTag handling and
> sibling recovery on EXCEED_LIMIT_LENGTH. That path is the remaining in-parser
> CHAR machinery after SPARK-59722 moved JSON to parse-as-STRING then
> assignAfterParse.
> XML is last on purpose. Do not combine this with the JSON or CSV PRs.
> Proposed change
> For from_xml and the XML datasource, use the same wrapping as SPARK-59722 /
> SPARK-60090:
> * Parse against a physical schema with CHAR/VARCHAR replaced by STRING
> (replaceCharVarcharWithStringAlways).
> * After parse, apply CharVarcharUtils.assignAfterParse so CHAR pad, overflow,
> and CHAR/VARCHAR map keys go through stringLengthCheck / MapFromArrays.
> * CHAR/VARCHAR-keyed maps then honor spark.sql.mapKeyDedupPolicy the same way
> JSON does after SPARK-59722, including exact repeated names.
> * Delete convertConstrainedMap and the MapType(CharType|VarcharType, ...)
> branch in convertField.
> * Remove applyTextParseSemantics from the XML STRING / empty-element
> converters.
> Attributes, rowTag, valueTag, and attributePrefix stay parser concerns. They
> must still be present on the parsed STRING row so assignment can pad or
> reject them. The goal is to drop CHAR-specific key draining and sibling
> recovery, not to rewrite the StAX walker.
> DUPLICATED_MAP_KEY from key assignment must still fail the query under
> PERMISSIVE (same as JSON/XML today).
> h3. User-facing change
> Only when spark.sql.charVarchar.standardSemantics.enabled is true:
> * CHAR/VARCHAR fields, attributes, and map keys still pad / length-check.
> * Overflow is a parse-mode error for the whole parsed value, not a null
> sibling with the rest of the row kept. That drops SPARK-59274 sibling
> recovery (e.g. mixed rowTag attributes where one overflows).
> * Exact repeated XML element names in a CHAR/VARCHAR-keyed map follow
> mapKeyDedupPolicy instead of the convertConstrainedMap last-wins rules.
> * Ordinary MAP<STRING, ...> duplicate behavior stays parser-historical.
> Document in the 4.3-to-4.4 SQL migration guide and on
> spark.sql.mapKeyDedupPolicy (today it already mentions XML CHAR keys).
> h3. Tests
> Port SPARK-59274 XML CHAR coverage in BasicCharVarcharTestSuite to post-parse
> expectations: padding, overflow, attributes, rowTag, nested maps, collated
> keys, FAILFAST, DUPLICATED_MAP_KEY vs LAST_WIN, STRING map duplicates
> unchanged. Expect whole-value nulls where tests currently keep valid siblings.
> h3. Related
> * SPARK-59274
> * SPARK-59722 (do not land XML on that PR)
> * SPARK-60090
> * the CSV follow-up (same helpers, no map keys)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
