[
https://issues.apache.org/jira/browse/SPARK-60102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60102:
-----------------------------------
Labels: pull-request-available (was: )
> Apply CSV CHAR/VARCHAR assignment after STRING parse
> -----------------------------------------------------
>
> Key: SPARK-60102
> URL: https://issues.apache.org/jira/browse/SPARK-60102
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Serge Rielau
> Priority: Major
> Labels: pull-request-available
>
> Context
>
> SPARK-59274 applies CHAR/VARCHAR assignment inside UnivocityParser via
> applyTextParseSemantics on each STRING field. SPARK-59722 (JSON) and
> SPARK-60090 (Hive TRANSFORM) instead parse as unbounded STRING, then wrap the
> result with CharVarcharUtils.assignAfterParse / stringLengthCheck.
>
> CSV is the next format. It has no map keys and little CHAR-specific recovery,
> so it is the check that assignAfterParse is not JSON-shaped.
>
> Do not wait on SPARK-60090 if those helpers are already on master via
> SPARK-59722 (replaceCharVarcharWithStringAlways,
> replaceCharVarcharWithStringInSchemaAlways, assignAfterParse). Do not combine
> this with XML.
>
> Proposed change
>
> For from_csv and the CSV datasource:
>
> * Build UnivocityParser converters from a physical schema in which every
> CHAR/VARCHAR is replaced by STRING (replaceCharVarcharWithStringAlways),
> including when first-class CHAR/VARCHAR types are enabled.
> * After a row is parsed, apply CharVarcharUtils.assignAfterParse (write-side
> pad / EXCEED_LIMIT_LENGTH).
> * Remove applyTextParseSemantics from the CSV STRING converter.
> * Skip CSV filter pushdown when the schema contains CHAR/VARCHAR, matching
> JacksonParser after SPARK-59722.
>
> Keep the wrapping reusable (not a JSON-only InternalRow shape). CSV has no
> MAP keys, so MapFromArrays / mapKeyDedupPolicy is out of scope.
>
> User-facing change
>
> Only when spark.sql.charVarchar.standardSemantics.enabled is true (still
> false by default):
>
> * CHAR padding and VARCHAR/CHAR overflow still apply to from_csv and CSV
> reads.
> * Overflow becomes a parse-mode error for the whole parsed row (PERMISSIVE:
> null; FAILFAST: EXCEED_LIMIT_LENGTH), matching JSON after SPARK-59722.
> SPARK-59274 currently fails the overflowing field only.
>
> Document in the 4.3-to-4.4 SQL migration guide.
> Tests
>
> Extend BasicCharVarcharTestSuite for from_csv + CSV datasource: padding,
> overflow under PERMISSIVE/FAILFAST, nested STRUCT/ARRAY, no change with the
> flag off. Keep SPARK-59274 CSV document-as-CHAR cases (padding vs delimiter).
>
> Related
>
> * SPARK-59274 (in-parser CHAR/VARCHAR for JSON/CSV/XML)
> * SPARK-59722 (JSON post-parse assignment)
> * SPARK-60090 (Hive TRANSFORM post-parse assignment)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
