johanl-db commented on code in PR #50315:
URL: https://github.com/apache/spark/pull/50315#discussion_r2005726489
##########
connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala:
##########
@@ -963,6 +964,28 @@ abstract class AvroSuite
}
}
+ test("SPARK-49082: Widening date to timestampNTZ in AvroDeserializer") {
+ withTempPath { tempPath =>
+ val datePath = s"$tempPath/date_data"
+ val dateDf =
+ Seq(LocalDate.of(2024, 1, 1),
+ LocalDate.of(2024, 1, 2),
+ LocalDate.of(1312, 2, 27),
+ LocalDate.of(-5877641, 6, 23),
+ LocalDate.of(5881580, 7, 11))
+ .toDF("col")
+ dateDf.write.format("avro").save(datePath)
+ checkAnswer(
+ spark.read.schema("col TIMESTAMP_NTZ").format("avro").load(datePath),
+ Seq(Row(LocalDateTime.of(2024, 1, 1, 0, 0)),
+ Row(LocalDateTime.of(2024, 1, 2, 0, 0)),
+ Row(LocalDateTime.of(1312, 2, 27, 0, 0)),
+ Row(LocalDateTime.of(-5877641, 6, 23, 0, 0)),
+ Row(LocalDateTime.of(5881580, 7, 11, 0, 0)))
Review Comment:
That's not great..
I think for now we can document that limitation. Iceberg allows widening
date -> timestamp without timezone and also calls this out:
https://iceberg.apache.org/spec/#schema-evolution
Hopefully very few customers are using date values outside `-290308-12-21
BCE` - `+294247-01-10 CE`
Moving forward, we'll probably want to address this by moving the error from
read time to write time, i.e. check that all values in the table fit the
timestamp range before applying the type change (which we should be able to
tell from stats only in most cases).
Note: this is a Delta issue really, so not issue for this spark PR in itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
