[
https://issues.apache.org/jira/browse/SPARK-60090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Rielau updated SPARK-60090:
---------------------------------
Description:
Parent / related: SPARK-59683 / PR
[#58936|https://github.com/apache/spark/pull/58936]
h3. What
Keep Hive {{TRANSFORM}} / {{MAP}} / {{REDUCE}} script I/O as unbounded
{{STRING}} (and JSON for complex types), the same as sources that have no
native CHAR/VARCHAR. After the script reader, apply Spark CHAR/VARCHAR
assignment ({{{}CharVarcharUtils.stringLengthCheck{}}}: pad {{{}CHAR{}}},
{{EXCEED_LIMIT_LENGTH}} on overflow) in a {{Project}} so the query result type
is the declared {{{}CHAR{}}}/{{{}VARCHAR{}}} (including nested, e.g.
{{{}MAP<CHAR(5), CHAR(10)>{}}}).
Do not teach the no-SerDe writers, Hive unwrappers, or SerDe schema rewrite to
understand CHAR/VARCHAR.
h3. Why
Default Spark (and pre first-class CHAR) already rewrote {{AS (c CHAR(5))}} to
{{{}STRING{}}}. No pad, no length check, no {{{}Cast{}}}. Hive TRANSFORM is
STRING-on-the-wire. First-class CHAR made {{CharType}} survive parse, so
SPARK-59683 built a deep codec inside the script/Hive path.
That mixed two problems:
# CHAR/VARCHAR assignment (pad / length) -- same as CSV/JSON/Parquet.
# TRANSFORM JSON / Hive wire issues that are not CHAR-specific (non-string map
keys, duplicate JSON keys, SerDe-dependent truncate vs fail). SPARK-59683 only
handled some of those on the CHAR path, which is why {{MAP<STRING, STRING>}} vs
{{MAP<STRING, CHAR(2)>}} diverged and DATE map keys can round-trip wrong.
TRANSFORM is Hive-compat, uncommon, documented in OSS Spark, not in Databricks
SQL {{{}SELECT{}}}. The CHAR epic already marked Hive TRANSFORM out of scope.
h3. Proposed plan
# Script exec output: recursively replace CHAR/VARCHAR with unbounded STRING
(historical {{replaceCharVarcharWithString}} behavior), including nested
maps/arrays/structs.
# Parse delimited / JSON as today for STRING and complex STRING-shaped types.
# Wrap with {{Project}} using {{stringLengthCheck}} (and CHAR pad) onto the
declared first-class types when
{{spark.sql.charVarchar.standardSemantics.enabled}} is true.
# Flag off: keep today's STRING result ({{{}AS (c CHAR(5)){}}} remains
{{STRING}} in the schema). No behavior change.
# Remove or do not land CHAR-specific logic in
{{BaseScriptTransformationExec}} converters, {{outputTypesForSerDe}} LazySimple
rewrite, and Hive {{unwrapperFor}} CHAR checks.
# JSON non-string map keys ({{{}MAP<INT, ...>{}}}, {{{}MAP<DATE, ...>{}}}):
out of scope. Separate ticket: reject unsafe key types, or a reversible codec
for all map key types. CHAR map keys are strings on the JSON wire and are
handled by assignment.
h3. Tests / acceptance
* Scalar {{AS (c CHAR(n))}} / {{{}VARCHAR(n){}}}: pad, overflow, null token;
result type is CHAR/VARCHAR with the flag on; STRING with the flag off.
* Nested: {{{}ARRAY<CHAR(n)>{}}}, {{{}STRUCT{}}}, {{MAP<CHAR(n), VARCHAR(m)>}}
via the {{{}Project{}}}, not a custom restorer.
* Hive SerDe and no-SerDe both go through the same Spark-side assignment
(script still STRING-shaped).
* Preserve-only ({{{}preserveCharVarcharTypeInfo{}}} without standard
semantics): no pad / no {{{}EXCEED_LIMIT_LENGTH{}}}.
* No CHAR-only change to map-key dedup vs {{{}MAP<STRING, STRING>{}}}.
h3. Note for implementer
SPARK-59683 / OSS PR 58936 should be superseded or stripped back to this model
rather than patched further (bind-at-parse flag, Hive unwrapper gating, JSON
restorer).
was:Our previous version of dealing with TRANSFORM... AS (c1 CHAR(5), c2
MAP<CHAR6>, ...>) was tryiong to teach hive about CHAR/VARCAHR. That is NOT
this featuires job. TRANSFORM just is another STRING only data source. We deal
with it on the Spark side by compensatin the date. YTyhsi greatly de-risks this
code path, while also giving us a clean API.
> TRANSFORM clause and CHAR: V2: Treat like any other STRING source
> -----------------------------------------------------------------
>
> Key: SPARK-60090
> URL: https://issues.apache.org/jira/browse/SPARK-60090
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: Serge Rielau
> Priority: Major
>
> Parent / related: SPARK-59683 / PR
> [#58936|https://github.com/apache/spark/pull/58936]
> h3. What
> Keep Hive {{TRANSFORM}} / {{MAP}} / {{REDUCE}} script I/O as unbounded
> {{STRING}} (and JSON for complex types), the same as sources that have no
> native CHAR/VARCHAR. After the script reader, apply Spark CHAR/VARCHAR
> assignment ({{{}CharVarcharUtils.stringLengthCheck{}}}: pad {{{}CHAR{}}},
> {{EXCEED_LIMIT_LENGTH}} on overflow) in a {{Project}} so the query result
> type is the declared {{{}CHAR{}}}/{{{}VARCHAR{}}} (including nested, e.g.
> {{{}MAP<CHAR(5), CHAR(10)>{}}}).
> Do not teach the no-SerDe writers, Hive unwrappers, or SerDe schema rewrite
> to understand CHAR/VARCHAR.
> h3. Why
> Default Spark (and pre first-class CHAR) already rewrote {{AS (c CHAR(5))}}
> to {{{}STRING{}}}. No pad, no length check, no {{{}Cast{}}}. Hive TRANSFORM
> is STRING-on-the-wire. First-class CHAR made {{CharType}} survive parse, so
> SPARK-59683 built a deep codec inside the script/Hive path.
> That mixed two problems:
> # CHAR/VARCHAR assignment (pad / length) -- same as CSV/JSON/Parquet.
> # TRANSFORM JSON / Hive wire issues that are not CHAR-specific (non-string
> map keys, duplicate JSON keys, SerDe-dependent truncate vs fail). SPARK-59683
> only handled some of those on the CHAR path, which is why {{MAP<STRING,
> STRING>}} vs {{MAP<STRING, CHAR(2)>}} diverged and DATE map keys can
> round-trip wrong.
> TRANSFORM is Hive-compat, uncommon, documented in OSS Spark, not in
> Databricks SQL {{{}SELECT{}}}. The CHAR epic already marked Hive TRANSFORM
> out of scope.
> h3. Proposed plan
> # Script exec output: recursively replace CHAR/VARCHAR with unbounded STRING
> (historical {{replaceCharVarcharWithString}} behavior), including nested
> maps/arrays/structs.
> # Parse delimited / JSON as today for STRING and complex STRING-shaped types.
> # Wrap with {{Project}} using {{stringLengthCheck}} (and CHAR pad) onto the
> declared first-class types when
> {{spark.sql.charVarchar.standardSemantics.enabled}} is true.
> # Flag off: keep today's STRING result ({{{}AS (c CHAR(5)){}}} remains
> {{STRING}} in the schema). No behavior change.
> # Remove or do not land CHAR-specific logic in
> {{BaseScriptTransformationExec}} converters, {{outputTypesForSerDe}}
> LazySimple rewrite, and Hive {{unwrapperFor}} CHAR checks.
> # JSON non-string map keys ({{{}MAP<INT, ...>{}}}, {{{}MAP<DATE, ...>{}}}):
> out of scope. Separate ticket: reject unsafe key types, or a reversible codec
> for all map key types. CHAR map keys are strings on the JSON wire and are
> handled by assignment.
> h3. Tests / acceptance
> * Scalar {{AS (c CHAR(n))}} / {{{}VARCHAR(n){}}}: pad, overflow, null token;
> result type is CHAR/VARCHAR with the flag on; STRING with the flag off.
> * Nested: {{{}ARRAY<CHAR(n)>{}}}, {{{}STRUCT{}}}, {{MAP<CHAR(n),
> VARCHAR(m)>}} via the {{{}Project{}}}, not a custom restorer.
> * Hive SerDe and no-SerDe both go through the same Spark-side assignment
> (script still STRING-shaped).
> * Preserve-only ({{{}preserveCharVarcharTypeInfo{}}} without standard
> semantics): no pad / no {{{}EXCEED_LIMIT_LENGTH{}}}.
> * No CHAR-only change to map-key dedup vs {{{}MAP<STRING, STRING>{}}}.
> h3. Note for implementer
> SPARK-59683 / OSS PR 58936 should be superseded or stripped back to this
> model rather than patched further (bind-at-parse flag, Hive unwrapper gating,
> JSON restorer).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
