yihua opened a new pull request, #20256: URL: https://github.com/apache/hudi/pull/20256
### Describe the issue this Pull Request addresses closes #20255 Data skipping used an expression index for any unary function over the index's source column. With an index on `lower(city)`, a filter `upper(city) = 'X'` was evaluated against the `lower` values and pruned every file holding matches (wrong result), and `length(city) = 9` failed with a `ClassCastException`. The catch-all `case expression: UnaryExpression => expression.child` came in with #12290 while a separate function-name check still guarded it; #12455 replaced that check with per-function option checks and dropped the function-name comparison, leaving the catch-all unguarded. ### Summary and Changelog Each match arm in `ExpressionIndexSupport.extractQueryAndLiterals` now also requires the index definition's function (`upper`, `lower`, `length`, `year`, `month`, `day`, `hour`, plus the existing format-based functions; `to_date` / `to_timestamp` without a format, which the optimizer turns into a `Cast`, match only their own index). Filters on other functions fall back to column stats. `TestExpressionIndex` adds a COW and MOR (with log files) test that `upper` and `length` filters return the same rows as with data skipping off (wrong / failing on master), and a test that filters on the indexed function still prune for `upper`, `length`, `year`, `hour` and `to_date`. Related: #17489 (an expression index matched by a direct column reference) touches the same method; this change does not cover that case. ### Impact Fixes wrong query results and query failures for tables with an expression index on a function of a column queried through a different function. Matches on the indexed function are unchanged. ### Risk Level low. With `spark.sql.timestampType=TIMESTAMP_NTZ`, a `to_timestamp(col)` filter (no format) becomes a cast to `TimestampNTZType` and no longer uses a `to_timestamp` expression index: results stay correct, only pruning is lost. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
