sepuri sai krishna created SPARK-60322:
------------------------------------------
Summary: Parquet and ORC filter pushdown drops rows for
comparisons with 0.0 or -0.0
Key: SPARK-60322
URL: https://issues.apache.org/jira/browse/SPARK-60322
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.3, 4.2.0, 3.5.3, 4.3.0, 5.0.0, 4.4.0
Reporter: sepuri sai krishna
Filter pushdown to Parquet and ORC skips rows when a FLOAT or DOUBLE column is
compared with a zero and the data holds the other zero. There is no error, the
rows are just missing. With the default configuration:
{code:sql}
CREATE TABLE p USING parquet AS SELECT id, -0.0D * id AS f FROM range(0, 1000,
1, 1);
SELECT count(*) FROM p WHERE f = 0; -- 0, expected 1000
CREATE TABLE o USING orc AS SELECT id, -0.0D * id AS f FROM range(1, 1000, 1,
1);
SELECT count(*) FROM o WHERE f = 0; -- 0, expected 999
SELECT count(*) FROM o WHERE f >= 0; -- 0, expected 999
SET spark.sql.parquet.filterPushdown=false;
SET spark.sql.orc.filterPushdown=false;
SELECT count(*) FROM p WHERE f = 0; -- 1000
SELECT count(*) FROM o WHERE f = 0; -- 999
{code}
Same results on 3.5.3, 4.0.3 and 4.2.0 (official binaries) and on master.
Spark treats 0.0 and -0.0 as equal ({{0.0D = -0.0D}} is true), but
{{ParquetFilters}} and {{OrcFilters}} push the literal down unchanged, and the
formats tell the two zeros apart:
* Parquet dictionary filter and bloom filter: a dictionary or bloom filter that
holds only -0.0 does not contain 0.0, so the row group is skipped. This is what
drops the Parquet rows above.
* ORC statistics: ORC compares with {{Double.compareTo}}, so a stripe or row
group whose min and max are -0.0 is skipped for {{f = 0.0}} or {{f >= 0.0}},
and one whose min and max are 0.0 for {{f = -0.0}} or {{f <= -0.0}}.
* Parquet statistics and column index: Parquet 1.17 and earlier write -0.0 as
the min and 0.0 as the max of a column chunk of zeros, so they do not skip
anything. Parquet 1.18 (master only, SPARK-58869) writes FLOAT and DOUBLE
columns with the IEEE 754 total order and exact bounds, so on master the
statistics and the column index skip rows as well, for files written with 1.18.
Affected predicates, with z = 0.0 or -0.0: {{f = z}}, {{f <=> z}}, {{f IN (z,
...)}}, {{f <= -0.0}}, {{f >= 0.0}}, and BETWEEN / OR built from them. A pushed
{{Not}} of {{f < z}}, {{f > z}} or {{f != z}} is wrong as well, since it
negates a comparison that keeps more values than the Spark predicate. The
optimizer usually rewrites such a {{NOT}} away, but not when that rule is
excluded.
Both FLOAT and DOUBLE, DSv1 and DSv2, vectorized and non-vectorized readers.
-0.0 is easy to get: {{-x}} or {{x * -1}} for x = 0.0, and data written by
other engines (pandas/numpy, Arrow, C++ writers) keeps the sign.
Workaround: {{spark.sql.parquet.filterPushdown=false}} /
{{spark.sql.orc.filterPushdown=false}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
