Aleksandar Trajković created SPARK-58825:
--------------------------------------------
Summary: Support TimestampNTZType as a JDBC partition column
Key: SPARK-58825
URL: https://issues.apache.org/jira/browse/SPARK-58825
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Aleksandar Trajković
h3. What
{{JDBCRelation}} restricts the {{partitionColumn}} option to numeric,
{{DateType}}, and
{{TimestampType}} columns (SPARK-22814). This extends that set to include
{{TimestampNTZType}}:
* {{verifyAndGetNormalizedPartitionColumn}} accepts {{TimestampNTZType}}.
* Bound parsing uses {{stringToTimestampWithoutTimeZone(_, allowTimeZone =
false)}}, so
{{lowerBound}}/{{upperBound}} are parsed as zoneless wall-clock values.
* Bound-to-string conversion formats the NTZ micros in UTC, so no
session-time-zone shift is
applied to the generated partition predicates.
Behavior for existing numeric/{{DateType}}/{{TimestampType}} partition columns
is unchanged.
h3. Why
More JDBC sources map zoneless database types (timestamp-without-time-zone) to
{{TimestampNTZType}}. Such a column is a valid, orderable partition column, but
today it is
rejected with {{invalidPartitionColumnTypeError}}.
h3. Example
{code:scala}
spark.read.format("jdbc")
.option("url", url)
.option("dbtable", "events")
.option("partitionColumn", "ts_ntz") // TimestampNTZType column
.option("lowerBound", "2018-07-06 00:00:00")
.option("upperBound", "2018-07-07 00:00:00")
.option("numPartitions", "4")
.load()
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
