[
https://issues.apache.org/jira/browse/SPARK-57783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57783:
-----------------------------
Description:
h2. Background
Umbrella: SPARK-57550 (Extend support for the TIME data type), SPIP SPARK-51162.
The TIME data type ({{TimeType}}) is being readied to enable by default
({{spark.sql.timeType.enabled}}). PySpark (SPARK-57565 / SPARK-57567 /
SPARK-57574 /
SPARK-57696) and Spark Connect (SPARK-57566) coverage are tracked, but there is
no
equivalent work for SparkR.
h2. Problem
SparkR's R <-> JVM serializer handles only the legacy {{java.sql}} datetime
types:
{code:scala}
// core/src/main/scala/org/apache/spark/api/r/SerDe.scala
import java.sql.{Date, Time, Timestamp}
{code}
{{writeObject}} matches {{java.sql.Date}} / {{java.sql.Time}} /
{{java.sql.Timestamp}}
but not {{java.time.LocalTime}}, which is the external type of {{TimeType}}. As
a result,
collecting or creating TIME columns through SparkR (non-Arrow path) falls
through to the
{{case _ => "Invalid type"}} branch, and there is no SparkR TIME test coverage.
h2. Scope
Rather than add full SparkR TIME support, document TIME as unsupported in
SparkR, matching
how Hive interop is already marked. Specifically:
* In {{docs/sql-ref-datatypes.md}}, mark {{TimeType}} as "Not supported" in the
R <-> Spark
type mapping table (the same way {{TimeType}} is shown as "Not supported" for
Hive).
* Optionally add a short note in the SparkR docs / migration guide.
h2. Acceptance
* The data types reference clearly states TIME is not supported in SparkR.
* No behavioral code change in SparkR is required by this ticket.
> Document the TIME data type as unsupported in SparkR
> ----------------------------------------------------
>
> Key: SPARK-57783
> URL: https://issues.apache.org/jira/browse/SPARK-57783
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Environment: h2. Background
> Umbrella: SPARK-57550 (Extend support for the TIME data type), SPIP
> SPARK-51162.
> The TIME data type ({{TimeType}}) is being readied to enable by default
> ({{spark.sql.timeType.enabled}}). PySpark (SPARK-57565 / SPARK-57567 /
> SPARK-57574 /
> SPARK-57696) and Spark Connect (SPARK-57566) coverage are tracked, but there
> is no
> equivalent work for SparkR.
> h2. Problem
> SparkR's R <-> JVM serializer handles only the legacy {{java.sql}} datetime
> types:
> {code:scala}
> // core/src/main/scala/org/apache/spark/api/r/SerDe.scala
> import java.sql.{Date, Time, Timestamp}
> {code}
> {{writeObject}} matches {{java.sql.Date}} / {{java.sql.Time}} /
> {{java.sql.Timestamp}}
> but not {{java.time.LocalTime}}, which is the external type of {{TimeType}}.
> As a result,
> collecting or creating TIME columns through SparkR (non-Arrow path) falls
> through to the
> {{case _ => "Invalid type"}} branch, and there is no SparkR TIME test
> coverage.
> h2. Scope
> Rather than add full SparkR TIME support, document TIME as unsupported in
> SparkR, matching
> how Hive interop is already marked. Specifically:
> * In {{docs/sql-ref-datatypes.md}}, mark {{TimeType}} as "Not supported" in
> the R <-> Spark
> type mapping table (the same way {{TimeType}} is shown as "Not supported"
> for Hive).
> * Optionally add a short note in the SparkR docs / migration guide.
> h2. Acceptance
> * The data types reference clearly states TIME is not supported in SparkR.
> * No behavioral code change in SparkR is required by this ticket.
> Reporter: Max Gekk
> Priority: Major
>
> h2. Background
> Umbrella: SPARK-57550 (Extend support for the TIME data type), SPIP
> SPARK-51162.
> The TIME data type ({{TimeType}}) is being readied to enable by default
> ({{spark.sql.timeType.enabled}}). PySpark (SPARK-57565 / SPARK-57567 /
> SPARK-57574 /
> SPARK-57696) and Spark Connect (SPARK-57566) coverage are tracked, but there
> is no
> equivalent work for SparkR.
> h2. Problem
> SparkR's R <-> JVM serializer handles only the legacy {{java.sql}} datetime
> types:
> {code:scala}
> // core/src/main/scala/org/apache/spark/api/r/SerDe.scala
> import java.sql.{Date, Time, Timestamp}
> {code}
> {{writeObject}} matches {{java.sql.Date}} / {{java.sql.Time}} /
> {{java.sql.Timestamp}}
> but not {{java.time.LocalTime}}, which is the external type of {{TimeType}}.
> As a result,
> collecting or creating TIME columns through SparkR (non-Arrow path) falls
> through to the
> {{case _ => "Invalid type"}} branch, and there is no SparkR TIME test
> coverage.
> h2. Scope
> Rather than add full SparkR TIME support, document TIME as unsupported in
> SparkR, matching
> how Hive interop is already marked. Specifically:
> * In {{docs/sql-ref-datatypes.md}}, mark {{TimeType}} as "Not supported" in
> the R <-> Spark
> type mapping table (the same way {{TimeType}} is shown as "Not supported"
> for Hive).
> * Optionally add a short note in the SparkR docs / migration guide.
> h2. Acceptance
> * The data types reference clearly states TIME is not supported in SparkR.
> * No behavioral code change in SparkR is required by this ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
