[
https://issues.apache.org/jira/browse/SPARK-57738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jubin Soni updated SPARK-57738:
-------------------------------
Description:
*What is the issue?*
{{ArrowVectorReader.applyDefault}} contains a guard that is supposed to reject
any attempt to read an unsupported target type over Spark Connect:
{{if (!UpCastRule.canUpCast(vectorDataType, targetDataType)) \{
throw new RuntimeException(
s"Reading '$targetDataType' values from a ${vector.getClass} instance is
not supported.")
}}}
SPARK-57303 updated {{UpCastRule.canUpCast}} to return {{true}} for lossless
widening within the timestamp family (e.g. {{{}TimestampType ->
TimestampLTZNanosType(p){}}}). As a side effect this guard now silently passes
for nanosecond timestamp targets, even though no Arrow vector encoding or
reader implementation exists for them. Execution then falls through the
{{vector match}} block to the generic catch-all:
{{case _ => throw new RuntimeException("Unsupported Vector Type: " +
vector.getClass)}}
This is a confusing, misleading error that gives no indication the problem is
the target type rather than the vector type. The SPARK-57303 commit message
explicitly acknowledges this regression: _"ArrowVectorReader's {{canUpCast}}
guard no longer fails fast on a micro-vector/nanos-target mismatch; whenever
nanos-over-Connect is implemented, that PR should add the reader and re-examine
this guard."_
----
*How to reproduce*
Call {{ArrowVectorReader.apply}} with a {{TimestampLTZNanosType}} or
{{TimestampNTZNanosType}} target against a micro-precision timestamp vector
(the vector type a Connect server sends for any LTZ timestamp column):
{{import org.apache.arrow.memory.RootAllocator
import org.apache.spark.sql.connect.client.arrow.ArrowVectorReader
import org.apache.spark.sql.types.\{TimestampLTZNanosType, TimestampType}
import org.apache.spark.sql.util.ArrowUtils
val allocator = new RootAllocator()
val vector = ArrowUtils.toArrowField("ts", TimestampType, nullable = true,
"UTC")
.createVector(allocator)
ArrowVectorReader(TimestampLTZNanosType(9), vector, "UTC")}}
----
*Actual behavior*
{{java.lang.RuntimeException: Unsupported Vector Type: class
org.apache.arrow.vector.TimeStampMicroTZVector}}
----
*Expected behavior*
{{java.lang.RuntimeException: Reading 'timestamp_ltz(9)' values over Spark
Connect is not yet supported.}}
----
*Proposed fix*
Add an explicit rejection on {{AnyTimestampNanoType}} in
{{ArrowVectorReader.applyDefault}} between the {{canUpCast}} guard and the
{{vector match}} block, in
{{{}sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowVectorReader.scala{}}}:
{{if (targetDataType.isInstanceOf[AnyTimestampNanoType]) \{
throw new RuntimeException(
s"Reading '$targetDataType' values over Spark Connect is not yet
supported.")
}}}
This guard should be removed when a proper Connect reader for nanosecond
timestamp types is implemented.
----
----
was:
After SPARK-57303 updated {{UpCastRule.canUpCast}} to return {{true}} for
lossless timestamp-family widening (e.g. {{{}TimestampType ->
TimestampLTZNanosType(p){}}}), the early-exit safety check in
{{ArrowVectorReader.applyDefault}} (line 83) no longer rejects requests to read
a nanosecond-precision timestamp column over Spark Connect.
If a client requests a {{TIMESTAMP_LTZ(p)}} or {{TIMESTAMP_NTZ(p)}} column with
{{{}p in [7,9]{}}}, the guard passes, but there is no matching {{case}} in the
{{vector match}} block for nanosecond Arrow vectors. Execution falls through to
the catch-all {{case _ => throw new RuntimeException("Unsupported Vector
Type")}} — a confusing crash instead of a clear "not supported" error.
The SPARK-57303 commit message explicitly acknowledges this:
_"ArrowVectorReader's {{canUpCast}} guard no longer fails fast on a
micro-vector/nanos-target mismatch; whenever nanos-over-Connect is implemented,
that PR should add the reader and re-examine this guard."_
*Fix:* Add an explicit rejection in {{applyDefault}} before the {{vector
match}} for {{{}TimestampLTZNanosType{}}}/{{{}TimestampNTZNanosType{}}} target
types, until a proper Connect reader is implemented. Relevant file:
{{sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowVectorReader.scala}}
> ArrowVectorReader guard no longer rejects unsupported nanosecond timestamp
> types over Spark Connect
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-57738
> URL: https://issues.apache.org/jira/browse/SPARK-57738
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.3.0
> Reporter: Jubin Soni
> Priority: Minor
> Labels: pull-request-available
>
> *What is the issue?*
> {{ArrowVectorReader.applyDefault}} contains a guard that is supposed to
> reject any attempt to read an unsupported target type over Spark Connect:
> {{if (!UpCastRule.canUpCast(vectorDataType, targetDataType)) \{
> throw new RuntimeException(
> s"Reading '$targetDataType' values from a ${vector.getClass} instance is
> not supported.")
> }}}
> SPARK-57303 updated {{UpCastRule.canUpCast}} to return {{true}} for lossless
> widening within the timestamp family (e.g. {{{}TimestampType ->
> TimestampLTZNanosType(p){}}}). As a side effect this guard now silently
> passes for nanosecond timestamp targets, even though no Arrow vector encoding
> or reader implementation exists for them. Execution then falls through the
> {{vector match}} block to the generic catch-all:
> {{case _ => throw new RuntimeException("Unsupported Vector Type: " +
> vector.getClass)}}
> This is a confusing, misleading error that gives no indication the problem is
> the target type rather than the vector type. The SPARK-57303 commit message
> explicitly acknowledges this regression: _"ArrowVectorReader's {{canUpCast}}
> guard no longer fails fast on a micro-vector/nanos-target mismatch; whenever
> nanos-over-Connect is implemented, that PR should add the reader and
> re-examine this guard."_
> ----
> *How to reproduce*
> Call {{ArrowVectorReader.apply}} with a {{TimestampLTZNanosType}} or
> {{TimestampNTZNanosType}} target against a micro-precision timestamp vector
> (the vector type a Connect server sends for any LTZ timestamp column):
> {{import org.apache.arrow.memory.RootAllocator
> import org.apache.spark.sql.connect.client.arrow.ArrowVectorReader
> import org.apache.spark.sql.types.\{TimestampLTZNanosType, TimestampType}
> import org.apache.spark.sql.util.ArrowUtils
> val allocator = new RootAllocator()
> val vector = ArrowUtils.toArrowField("ts", TimestampType, nullable = true,
> "UTC")
> .createVector(allocator)
> ArrowVectorReader(TimestampLTZNanosType(9), vector, "UTC")}}
> ----
> *Actual behavior*
> {{java.lang.RuntimeException: Unsupported Vector Type: class
> org.apache.arrow.vector.TimeStampMicroTZVector}}
> ----
> *Expected behavior*
> {{java.lang.RuntimeException: Reading 'timestamp_ltz(9)' values over Spark
> Connect is not yet supported.}}
> ----
> *Proposed fix*
> Add an explicit rejection on {{AnyTimestampNanoType}} in
> {{ArrowVectorReader.applyDefault}} between the {{canUpCast}} guard and the
> {{vector match}} block, in
> {{{}sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowVectorReader.scala{}}}:
> {{if (targetDataType.isInstanceOf[AnyTimestampNanoType]) \{
> throw new RuntimeException(
> s"Reading '$targetDataType' values over Spark Connect is not yet
> supported.")
> }}}
> This guard should be removed when a proper Connect reader for nanosecond
> timestamp types is implemented.
> ----
> ----
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
