andygrove opened a new issue, #36: URL: https://github.com/apache/datafusion-iceberg/issues/36
### Describe the bug `to_iceberg_predicate` unwraps every `Expr::Cast` except casts to `Date32`/`Date64` and pushes the comparison down against the uncast column ([expr_to_predicate.rs#L148-L157](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/physical_plan/expr_to_predicate.rs#L148-L157)). That is only safe for lossless, order-preserving casts. For other casts, the pushed predicate isn't implied by the original filter. Iceberg uses it to prune files and filter rows, so matching rows are lost. Because pushdown is `Inexact`, DataFusion re-applies the original filter afterwards, but it can't bring those rows back. When the literal can't be converted to the column's type, Iceberg fails to bind the predicate and the whole query errors. DataFusion's type coercion puts these casts on the column side in ordinary queries, for example a REAL column compared with a DOUBLE literal, or an INT column compared with a decimal literal. ### To Reproduce Results are compared with the same rows in a `MemTable`: | Column and rows | Filter | Pushed predicate | Iceberg result | Expected | |---|---|---|---|---| | `x REAL` = 0.1, 0.2 | `x > 0.1` | `x > 0.1` | `0.2` | `0.1, 0.2` | | `d DOUBLE` = 0.1, 0.2 | `CAST(d AS REAL) = CAST(0.1 AS REAL)` | `d = 0.10000000149011612` | no rows | `0.1` | | `i INT` = 9, 10, 11 | `i > 9.5` | `i > 9.5` | error: `Can't convert datum from double type to int type` | `10, 11` | | `f DOUBLE` = 5.7, 5.0, 6.2 | `CAST(f AS INT) = 5` | `f = 5` | error: `Can't convert datum from int type to double type` | `5.7, 5.0` | | `s STRING` = '05', '5' | `CAST(s AS INT) = 5` | `s = 5` | error: `Can't convert datum from int type to string type` | `05, 5` | | `i INT` = 9, 10, 11 | `NOT isnan(i)` | `NOT (i IS NAN)` | error: `Expecting floating point type, but found int` | `9, 10, 11` | The first row is the one users are most likely to hit. DataFusion evaluates `x > 0.1` as `CAST(x AS DOUBLE) > 0.1`, which is true for `0.1f32`. The pushed `x > 0.1` is bound as a float comparison, which is false for that row. ### Expected behavior Push a cast column down only when the cast is known to be lossless and order-preserving, such as widening integer casts. Otherwise leave the filter to DataFusion. ### Additional context - The `isnan` path strips casts the same way: [expr_to_predicate.rs#L275-L284](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/physical_plan/expr_to_predicate.rs#L275-L284). - apache/iceberg-rust#976 fixed this only for casts to Date. - Casts on the literal side are safe to unwrap and should keep working, e.g. `ts >= CAST('2023-01-05' AS TIMESTAMP)` (apache/iceberg-rust#942). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
