hudi-agent commented on code in PR #20295:
URL: https://github.com/apache/hudi/pull/20295#discussion_r4255627782
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/ColumnStatsProbe.java:
##########
@@ -60,6 +60,14 @@ public class ColumnStatsProbe implements Serializable {
* @return true if the index row should be considered as a candidate
*/
public boolean test(RowData indexRow, RowType.RowField[] queryFields) {
+ // Retain candidates with missing column statistics rather than treating
unknown values as nulls.
+ for (int i = 0; i < queryFields.length; i++) {
+ // The row starts with file name and value count, followed by (min, max,
null count) per column.
+ // Skip the two leading fields and i triples, then use offset 2 to check
this column's null count.
+ if (indexRow.isNullAt(2 + i * 3 + 2)) {
Review Comment:
🤖 non-blocking: Because this returns before any evaluator runs, a file
missing stats for any referenced column also stops being pruned on columns
whose stats are complete. For example, with `age > 100 AND name = 'x'`, no file
that predates the index expansion can be skipped on `age` anymore, though it
could before. Would it make sense to treat unknown stats as "may match" at the
leaf level instead, e.g. leave the column out of `columnStatsMap` and have
`getColumnStats` callers return true when it's absent? That way AND can still
prune on the known columns.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
