[
https://issues.apache.org/jira/browse/SPARK-60301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60301:
-----------------------------------
Labels: pull-request-available (was: )
> Spark doesn't respect session timezone when casting nested variant to string
> ----------------------------------------------------------------------------
>
> Key: SPARK-60301
> URL: https://issues.apache.org/jira/browse/SPARK-60301
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.1.3, 4.0.4
> Reporter: Chenhao Li
> Priority: Major
> Labels: pull-request-available
>
> **Top-level variant** (`cast(v as string)`): Spark uses the cast's time zone.
> In `Cast.scala`, a cast from `VariantType` calls `VariantGet.cast(v, to,
> castArgs)`. `castArgs` is a `VariantCastArgs` that carries the cast's
> `timeZoneId` and `zoneId`.
> **Variant nested in a struct, array, or map** (`cast(struct(v) as string)`):
> Spark ignores the cast's time zone and always uses UTC.
> - In `ToStringBase.scala`, `castToString` sends the leaf type to
> `TypeApiOps(from, zoneId)`. `TypeApiOps.apply` has no `VariantType` case, so
> it returns `None`.
> - `castToStringDefault` has no `VariantType` case either. It falls through to
> `case _ => o => UTF8String.fromString(o.toString)`.
> - The codegen path, `castToStringCode`, also has no `VariantType` case. It
> falls through to `UTF8String.fromString(String.valueOf($c))`.
> - Both paths call `VariantVal.toString()`, which is `toJson(ZoneOffset.UTC)`.
> - `Variant.toJson` prints a timestamp with `.atZone(zoneId)` and a `+HH:MM`
> offset. Here `zoneId` is UTC.
> The nested behavior comes from a fall-through to `toString`, not from a
> deliberate choice.
> **Example:** the session time zone is `America/Los_Angeles`, and a variant
> holds the instant `2020-01-01 00:00:00 UTC`.
> - `cast(named_struct('a', v) as string)` prints the timestamp in UTC,
> approximately `\{2020-01-01 00:00:00+00:00}`.
> - A top-level `cast(v as string)` uses the cast's time zone (here LA).
> The same variant value therefore prints with a different offset at top level
> than when it is nested.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
