darshil929 opened a new issue, #101:
URL: https://github.com/apache/datafusion-iceberg/issues/101
### Describe the bug
Some filters on `double` and `float` columns return fewer rows from an
Iceberg table than DataFusion returns for the same data. The missing rows are
the ones that hold NaN or -0.0. The query succeeds, but those rows aren't in
the result.
The two sides compare these values differently:
* DataFusion treats NaN as equal to itself and greater than any other value,
and treats -0.0 as equal to 0.0. `SELECT CAST('NaN' AS DOUBLE) > 1.0,
CAST('NaN' AS DOUBLE) = CAST('NaN' AS DOUBLE), CAST('-0.0' AS DOUBLE) = 0.0`
returns true for all three.
* iceberg-rust's `InclusiveMetricsEvaluator` decides that a data file
holding only NaN can't match `>`, `>=`, `=` or `IN`, and NaN is left out of a
file's upper bound. iceberg-rust also orders floats with `total_cmp`, so to it
-0.0 and 0.0 are different values.
`scalar_value_to_datum` passes float literals through as `Datum::double`,
and the comparison reaches the scan unchanged. So the scan skips files that
contain rows DataFusion would match. Those files are never read, and filtering
again in DataFusion can't bring the rows back.
### To Reproduce
Using the sqllogictest engine, with one INSERT per data file:
```sql
CREATE TABLE default.default.t (id INT NOT NULL, x DOUBLE);
INSERT INTO default.default.t VALUES (1, CAST('NaN' AS DOUBLE));
INSERT INTO default.default.t VALUES (2, 0.5), (3, CAST('NaN' AS DOUBLE));
INSERT INTO default.default.t VALUES (4, 2.0);
INSERT INTO default.default.t VALUES (5, CAST('-0.0' AS DOUBLE));
INSERT INTO default.default.t VALUES (6, 0.0);
CREATE TABLE m AS SELECT * FROM default.default.t;
```
`SELECT id FROM <table> WHERE <filter>` returns:
| Filter | Iceberg table `t` | MemTable copy `m` |
|---|---|---|
| `x > 1.0` | 4 | 1, 3, 4 |
| `x = CAST('NaN' AS DOUBLE)` | none | 1, 3 |
| `x = 0.0` | 6 | 5, 6 |
| `x >= 0.0` | 2, 3, 4, 6 | 1, 2, 3, 4, 5, 6 |
The same table returns 1, 3, 4 for `x + 0 > 1.0`, which isn't pushed down.
Which rows go missing also depends on how they're laid out in files. Row 3
is skipped because its file's upper bound is 0.5. The same NaN in a file with
2.0 comes back.
### Expected behavior
`t` returns the same rows as `m`.
I can think of two ways to get there:
1. Stop pushing down comparisons and `IN` lists that have a float literal.
That's the simplest, but float columns lose file pruning.
2. Push a looser predicate that still covers every row DataFusion could
match, and leave the exact check to DataFusion. For example, `x > c` becomes `x
> c OR x IS NAN` and `x = NaN` becomes `x IS NAN`. Zero literals would cover
both signs: `x = 0.0` becomes `x = -0.0 OR x = 0.0`, `x >= 0.0` becomes `x >=
-0.0`, and `x <= -0.0` becomes `x <= 0.0`.
Let me know which you'd prefer.
### Additional context
* `<`, `!=`, `NOT IN`, `isnan(x)` and `NOT isnan(x)` return the same rows on
both sides.
* `NOT (x <= 1.0)` hits this too, since it's simplified to `x > 1.0` before
pushdown.
* `REAL` columns are affected when the literal is also `REAL`, for example
`x > CAST(1 AS REAL)`. With a `DOUBLE` literal the column gets a cast, and
since #91 that comparison isn't pushed down.
* iceberg-rust main has the same NaN handling and float ordering, so
updating the iceberg-rust dependency won't fix this.
* This is the same kind of problem as #36 and #44: the pushed predicate
rejects rows that the filter keeps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
