mzabaluev opened a new issue, #27521: URL: https://github.com/apache/datafusion/issues/27521
### Describe the bug `get_field` on a `Map` reports `ExpressionPlacement::MoveTowardsLeafNodes` placement, but it can fail at execution. Leaf expression pushdown evaluates it on rows that a `CASE` branch or a filter would exclude, so a query that should succeed returns an error. This is a built-in instance of the undocumented requirement described in #27487: expressions with `MoveTowardsLeafNodes` placement must be safe to evaluate on rows that an enclosing filter or conditional would exclude. ### To Reproduce ```sql create table mt as values (1, map(['a', 'b'], [10, 20])), (2, map(['c'], [30])); SELECT CASE WHEN column1 < 0 THEN column2[7] ELSE 0 END FROM mt WHERE column1 <> 5; -- DataFusion error: Execution error: The key type Int64 does not match the map key type Utf8 ``` No row satisfies `column1 < 0`, so the lookup should never be evaluated. With `SET datafusion.optimizer.enable_leaf_expression_pushdown = false`, the query returns `0, 0`. With the default settings, the lookup is taken out of the `THEN` branch and evaluated below the filter for every row: ``` logical_plan 01)Projection: CASE WHEN mt.column1 < Int64(0) THEN __datafusion_extracted_1 ELSE Int64(0) END AS ... 02)--Filter: mt.column1 != Int64(5) 03)----Projection: get_field(mt.column2, Int64(7)) AS __datafusion_extracted_1, mt.column1 04)------TableScan: mt projection=[column1, column2] ``` Without the `WHERE` clause nothing is extracted, and the same `CASE` returns `0, 0`. So whether the query fails depends on an unrelated filter being present. ### Expected behavior One of these, preferably the first: - The planner rejects `column2[7]` with a type error, or coerces the key to the map's key type. Then the query either fails consistently at planning or succeeds, whichever branches are taken. - If a mismatched key is left to execution, the lookup is not evaluated speculatively, and the query returns `0, 0` as it does with leaf pushdown disabled. ### Additional context The error does not depend on the data. `map_lookup` (`datafusion/functions/src/utils.rs`) fails when the lookup key's type doesn't match the map's key type, and when a nested key has a different structure than the map's keys (through `make_comparator`). It fails on any batch it is evaluated on, whatever the values are. The planner could reject this, but it doesn't: `GetFieldFunc::return_field_from_args` (`datafusion/functions/src/core/getfield.rs`) resolves the value field of a `Map` without checking the key argument's type, so `column2[7]` plans successfully. The mismatch stays hidden as long as no row reaches the expression, and leaf pushdown removes exactly that guard. Possible fixes: - Check the key type in `get_field`'s `return_field_from_args`, or coerce the key to the map's key type, so a mismatched key is a planning error. As far as I can tell, map lookups then cannot fail at execution, and `MoveTowardsLeafNodes` would be safe for them. - Until then, `get_field` could return `KeepInPlace` when the access path reaches into a `Map`. `placement()` only receives the argument placements, not their types, so this needs either a type-aware placement API or a check where the types are known. #26589 currently works around this by not precomputing field accesses that reach into a `Map` inside `CASE`. That only covers the physical `CASE` evaluation, not the logical extraction shown above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
