cshuo opened a new pull request, #20295: URL: https://github.com/apache/hudi/pull/20295
### Describe the issue this Pull Request addresses Closes https://github.com/apache/hudi/issues/20289. After expanding the Flink column stats index from `age` to `age,name`, older files can still lack `name` statistics even though they contain non-null names. A query such as `age >= 0 AND name = 'alice'` then incorrectly skips those files and returns an empty result. The existing fallback only recovers files entirely absent from the requested index projection. ### Summary and Changelog - Represent missing column statistics as `(null, null, null)` instead of treating the column as entirely null. - Retain candidates with a missing null-count before evaluating predicates. Complete statistics, including known all-null columns, continue through the existing evaluator. - Add probe tests distinguishing missing statistics from all-null and nonmatching data, an index expansion regression test, and a SQL integration test comparing data skipping disabled/enabled. ### Impact Prevents missing rows when indexed columns have incomplete coverage across files. Files with incomplete statistics are conservatively scanned, which can increase reads even when another predicate could exclude them. No new configuration, public API, or persisted storage-format changes. ### Risk Level Low. The change conservatively retains candidates with missing statistics and leaves evaluation of complete statistics unchanged. Validation: 32 targeted JUnit tests passed using Flink 1.20.1, covering `TestColumnStatsProbe`, `TestColumnStatsIndex`, `TestExpressionEvaluators`, and `ITTestHoodieDataSource.testDataSkippingWithMissingColumnStatsAfterIndexExpansion`. The SQL integration test failed before the fix and passed afterward. Relevant sources were compiled against locally built dependencies; a full Maven build and the complete integration suite were not run. `git diff --check` passed. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
