[
https://issues.apache.org/jira/browse/SPARK-60326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60326:
-----------------------------------
Labels: pull-request-available (was: )
> ORC filter pushdown drops rows for TIMESTAMP literals with sub-millisecond
> digits
> ---------------------------------------------------------------------------------
>
> Key: SPARK-60326
> URL: https://issues.apache.org/jira/browse/SPARK-60326
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.0.3
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: pull-request-available
>
> ORC filter pushdown drops rows when a TIMESTAMP filter literal has
> sub-millisecond digits. With the default configuration:
> {code:sql}
> CREATE TABLE t USING orc AS SELECT 1 AS id, TIMESTAMP'2024-06-01
> 12:00:00.123456' AS ts;
> SELECT count(*) FROM t WHERE ts = TIMESTAMP'2024-06-01 12:00:00.123456'; --
> 0, expected 1
> SELECT count(*) FROM t WHERE ts <= TIMESTAMP'2024-06-01 12:00:00.123456'; --
> 0, expected 1
> SELECT count(*) FROM t WHERE ts >= TIMESTAMP'2024-06-01 12:00:00.123456'; --
> 1
> CREATE TABLE m USING orc AS SELECT id, TIMESTAMP'2024-06-01 12:00:00.123' AS
> ts FROM range(10);
> SELECT count(*) FROM m WHERE ts != TIMESTAMP'2024-06-01 12:00:00.123456'; --
> 0, expected 10
> SELECT count(*) FROM m WHERE ts < TIMESTAMP'2024-06-01 12:00:00.123001'; --
> 0, expected 10
> SET spark.sql.orc.filterPushdown=false; -- all of the above are then correct
> {code}
> Same for {{<=>}}, {{<}} and {{IN}}, on 4.0.3, 4.2.0 (official binaries) and
> master, with the V1 and V2 readers, the vectorized reader on and off, and
> {{spark.sql.orc.impl=hive}}. The nanosecond timestamp types on master
> (SPARK-57823) lose rows in the same way.
> The {{SearchArgument}} that Spark builds has the exact literal, and the ORC
> file statistics are exact too: reading the file with the ORC library and that
> {{SearchArgument}} returns the row. But Spark does not give the
> {{SearchArgument}} to the reader. {{OrcFileFormat}} and
> {{OrcPartitionReaderFactory}} store it in the Hadoop conf with
> {{OrcInputFormat.setSearchArgument}}, which serializes it with Kryo, and the
> reader decodes it with {{OrcInputFormat.buildOptions}}. Kryo's default
> serializer for {{java.sql.Timestamp}} only keeps milliseconds, so the reader
> gets {{2024-06-01 12:00:00.123}}:
> * {{ts = .123456}} and {{ts <= .123456}} become {{ts = .123}} and {{ts <=
> .123}}, which skip a stripe or row group whose minimum is {{.123456}};
> * {{ts != .123456}} becomes {{ts != .123}}, which skips a stripe or row group
> whose values are all {{.123}}.
> {{>}} and {{>=}} only get wider. A literal in the middle of a row group gives
> correct results, so it only shows when the literal is within a millisecond of
> the minimum or maximum of a row group, stripe or file, e.g. a file of one
> value, or a lookup of the first event of a file.
> Spark builds the {{SearchArgument}} on the executor right before it creates
> the reader, so it can pass it to {{Reader.Options.searchArgument}} directly.
> With {{spark.sql.orc.impl=hive}}, Hive's {{OrcInputFormat}} reads the
> {{SearchArgument}} from the conf ({{sarg.pushdown}}) with Kryo, so that one
> has to be handled differently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
