Yaoxuan Wu created SPARK-60152:
----------------------------------
Summary: ORC filter pushdown drops rows with a TIMESTAMP before
1582-10-15
Key: SPARK-60152
URL: https://issues.apache.org/jira/browse/SPARK-60152
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0, 5.0.0
Reporter: Yaoxuan Wu
{code:java}
import tempfile
from pyspark.sql import SparkSession
spark = SparkSession.builder.config("spark.sql.session.timeZone",
"UTC").getOrCreate()
path = tempfile.mkdtemp()
spark.sql("SELECT TIMESTAMP'0001-01-02 00:00:00' AS
t").write.mode("overwrite").orc(path)
spark.read.orc(path).createOrReplaceTempView("o")spark.sql("SELECT cast(t AS
STRING) FROM o").collect()
# [Row(t='0001-01-02 00:00:00')]: the value round-tripsspark.sql("SELECT
count(*) FROM o WHERE t = TIMESTAMP'0001-01-02 00:00:00'").first()[0]
# expected: 1
# Spark 4.2.0: 0
spark.sql("SELECT count(*) FROM o WHERE t <= TIMESTAMP'0001-01-02
00:00:00'").first()[0]
# expected: 1
# Spark 4.2.0: 0
spark.sql("SELECT count(*) FROM o WHERE t >= TIMESTAMP'0001-01-02
00:00:00'").first()[0]
# 1 {code}
With {{spark.sql.orc.filterPushdown=false}} all three return 1. Same on 3.5.8,
4.0.0, 4.1.1, 4.2.0 and master, with the JVM time zone UTC or
America/Los_Angeles, and with the vectorized reader on or off.
||Value||Predicates that return 0 rows||
|TIMESTAMP '0001-01-02 00:00:00'|{{{}={}}}, {{<=}}|
|TIMESTAMP '1500-01-01 00:00:00'|{{{}={}}}, {{>=}}|
|TIMESTAMP '1582-10-14 23:00:00'|{{=}}|
|TIMESTAMP '1582-10-16 00:00:00', DATE '0001-01-02', TIMESTAMP_NTZ '0001-01-02
00:00:00'|none|
*Cause* (checked with the ORC Java API through py4j):
* Spark writes TIMESTAMP values rebased to the hybrid Julian/Gregorian
calendar ({{{}OrcSerializer{}}} uses {{{}DateTimeUtils.toJavaTimestamp{}}}),
and builds the pushed literal the same way
({{{}OrcFilters.castLiteralValue{}}}). The literal's {{getTime()}} equals the
file statistics' {{{}getMinimum().getTime(){}}}.
* ORC's row-group pruning, {{{}RecordReaderImpl.evaluatePredicateProto{}}},
reads the statistics with {{{}ColumnStatisticsImpl.deserialize(null,
statsProto, writerUsedProlepticGregorian, true){}}}. The {{true}} converts
min/max to the proleptic Gregorian calendar (2 days later in year 1, 9 days
earlier in 1500). The hybrid literal is then outside [min, max].
* Reading the file with ORC directly and the same SearchArgument: Spark's
hybrid literal reads 0 rows, a proleptic literal ({{{}new Timestamp(micros /
1000){}}}) reads 1 row. ORC 1.9.8 and 2.3.0 behave the same.
* DATE works because ORC compares dates as local dates. TIMESTAMP_NTZ is
pushed as a long.
A fix on the Spark side could push TIMESTAMP literals without the hybrid
rebase, or skip pushdown for literals before 1582-10-15.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
