Yibo Dong created SPARK-59605:
---------------------------------
Summary: ORC filter pushdown fails on an `IN` list containing
`NULL`
Key: SPARK-59605
URL: https://issues.apache.org/jira/browse/SPARK-59605
Project: Spark
Issue Type: Bug
Components: Bug, Optimizer
Affects Versions: 5.0.0
Reporter: Yibo Dong
h2. What happened
ORC filter pushdown fails when an {{IN}} predicate contains a {{NULL}} literal.
The query executes correctly when {{spark.sql.orc.filterPushdown}} is disabled,
but fails with a {{NullPointerException}} when filter pushdown is enabled.
h2. How to reproduce
Run the following SQL:
{code:sql}
DROP TABLE IF EXISTS T;
CREATE TABLE T USING ORC AS
SELECT id, label
FROM VALUES
(NULL, 'null'),
(1, 'a'),
(2, 'b'),
(3, 'c') AS t(id, label);
SELECT id, label
FROM T
WHERE id IN (1, 3, NULL)
ORDER BY id, label;
{code}
Run it with ORC filter pushdown enabled:
{code:bash}
spark-sql
--master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.adaptive.enabled=false
--conf spark.sql.shuffle.partitions=1
--conf spark.sql.orc.filterPushdown=true
-f repro.sql
{code}
Then run the same query with ORC filter pushdown disabled:
{code:bash}
spark-sql
--master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.adaptive.enabled=false
--conf spark.sql.shuffle.partitions=1
--conf spark.sql.orc.filterPushdown=false
-f repro.sql
{code}
h2. Expected result
{{id IN (1, 3, NULL)}} should match the rows with {{id = 1}} and {{id = 3}}.
Both configurations should return:
{code}
1 a
3 c
{code}
h2. Actual result
With {{spark.sql.orc.filterPushdown=false}}, Spark returns the expected rows:
{code}
1 a
3 c
{code}
With {{spark.sql.orc.filterPushdown=true}}, the query fails while constructing
the ORC filter:
{code}
org.apache.spark.SparkException: [FAILED_READ_FILE.NO_HINT]
Encountered error while reading file ...
Caused by: java.lang.NullPointerException:
Cannot invoke "java.lang.Number.longValue()" because "value" is null
at
org.apache.spark.sql.execution.datasources.orc.OrcFilters$.castLiteralValue(OrcFilters.scala:167)
at
org.apache.spark.sql.execution.datasources.orc.OrcFilters$.buildLeafSearchArgument(OrcFilters.scala:284)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
