[
https://issues.apache.org/jira/browse/SPARK-57738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jubin Soni updated SPARK-57738:
-------------------------------
Affects Version/s: 4.3.0
(was: 4.0.3)
> ArrowVectorReader guard no longer rejects unsupported nanosecond timestamp
> types over Spark Connect
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-57738
> URL: https://issues.apache.org/jira/browse/SPARK-57738
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.3.0
> Reporter: Jubin Soni
> Priority: Minor
>
> After SPARK-57303 updated {{UpCastRule.canUpCast}} to return {{true}} for
> lossless timestamp-family widening (e.g. {{{}TimestampType ->
> TimestampLTZNanosType(p){}}}), the early-exit safety check in
> {{ArrowVectorReader.applyDefault}} (line 83) no longer rejects requests to
> read a nanosecond-precision timestamp column over Spark Connect.
> If a client requests a {{TIMESTAMP_LTZ(p)}} or {{TIMESTAMP_NTZ(p)}} column
> with {{{}p in [7,9]{}}}, the guard passes, but there is no matching {{case}}
> in the {{vector match}} block for nanosecond Arrow vectors. Execution falls
> through to the catch-all {{case _ => throw new RuntimeException("Unsupported
> Vector Type")}} — a confusing crash instead of a clear "not supported" error.
> The SPARK-57303 commit message explicitly acknowledges this:
> _"ArrowVectorReader's {{canUpCast}} guard no longer fails fast on a
> micro-vector/nanos-target mismatch; whenever nanos-over-Connect is
> implemented, that PR should add the reader and re-examine this guard."_
> *Fix:* Add an explicit rejection in {{applyDefault}} before the {{vector
> match}} for {{{}TimestampLTZNanosType{}}}/{{{}TimestampNTZNanosType{}}}
> target types, until a proper Connect reader is implemented. Relevant file:
> {{sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowVectorReader.scala}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
