Max Gekk created SPARK-51661:
--------------------------------
Summary: Partitions discovery of TIME column values
Key: SPARK-51661
URL: https://issues.apache.org/jira/browse/SPARK-51661
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.1.0
Reporter: Max Gekk
Assignee: Max Gekk
Modify the partition discovery mechanism, and support inferring of the TIME
data type if partition values match to the time pattern *HH:mm:ss[.SSSSSS]*.
Currently, for the time values like "10:11:12.123" and "23:59:59.1", the
inferred type is STRING. For example:
{code:scala}
scala> sql("SELECT time'12:00' AS t, 0 as
id").write.partitionBy("t").parquet("/Users/maxim.gekk/tmp/time_parquet2")
scala> spark.read.parquet("/Users/maxim.gekk/tmp/time_parquet2").printSchema()
root
|-- id: integer (nullable = true)
|-- t: string (nullable = true)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
