Jubin Soni created SPARK-57738:
----------------------------------
Summary: ArrowVectorReader guard no longer rejects unsupported
nanosecond timestamp types over Spark Connect
Key: SPARK-57738
URL: https://issues.apache.org/jira/browse/SPARK-57738
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 4.0.3
Reporter: Jubin Soni
After SPARK-57303 updated {{UpCastRule.canUpCast}} to return {{true}} for
lossless timestamp-family widening (e.g. {{{}TimestampType ->
TimestampLTZNanosType(p){}}}), the early-exit safety check in
{{ArrowVectorReader.applyDefault}} (line 83) no longer rejects requests to read
a nanosecond-precision timestamp column over Spark Connect.
If a client requests a {{TIMESTAMP_LTZ(p)}} or {{TIMESTAMP_NTZ(p)}} column with
{{{}p in [7,9]{}}}, the guard passes, but there is no matching {{case}} in the
{{vector match}} block for nanosecond Arrow vectors. Execution falls through to
the catch-all {{case _ => throw new RuntimeException("Unsupported Vector
Type")}} — a confusing crash instead of a clear "not supported" error.
The SPARK-57303 commit message explicitly acknowledges this:
_"ArrowVectorReader's {{canUpCast}} guard no longer fails fast on a
micro-vector/nanos-target mismatch; whenever nanos-over-Connect is implemented,
that PR should add the reader and re-examine this guard."_
*Fix:* Add an explicit rejection in {{applyDefault}} before the {{vector
match}} for {{{}TimestampLTZNanosType{}}}/{{{}TimestampNTZNanosType{}}} target
types, until a proper Connect reader is implemented. Relevant file:
{{sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowVectorReader.scala}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
