cshuo commented on code in PR #20295:
URL: https://github.com/apache/hudi/pull/20295#discussion_r4258718670
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/ColumnStatsProbe.java:
##########
@@ -60,6 +60,14 @@ public class ColumnStatsProbe implements Serializable {
* @return true if the index row should be considered as a candidate
*/
public boolean test(RowData indexRow, RowType.RowField[] queryFields) {
+ // Retain candidates with missing column statistics rather than treating
unknown values as nulls.
+ for (int i = 0; i < queryFields.length; i++) {
+ // The row starts with file name and value count, followed by (min, max,
null count) per column.
+ // Skip the two leading fields and i triples, then use offset 2 to check
this column's null count.
+ if (indexRow.isNullAt(2 + i * 3 + 2)) {
+ return true;
Review Comment:
Added coverage in 04cabae59bcb documenting the conservative retention when
name stats are missing, even though age alone excludes the file. Let’s leave
finer-grained pruning to a follow-up, accounting for `NOT` semantics.
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/stats/TestColumnStatsIndex.java:
##########
@@ -164,6 +172,60 @@ void testReadColumnStatsForCandidatePartitions(String
partition) throws Exceptio
}
}
+ @Test
+ void testCandidateFilesWithMissingColumnStatsAfterIndexExpansion() throws
Exception {
+ final String path = tempFile.getAbsolutePath();
+ Configuration conf = TestConfigurations.getDefaultConf(path);
+ conf.set(FlinkOptions.TABLE_TYPE, "COPY_ON_WRITE");
+ conf.set(FlinkOptions.METADATA_ENABLED, true);
+ conf.set(FlinkOptions.READ_DATA_SKIPPING_ENABLED, true);
+
conf.setString(HoodieMetadataConfig.ENABLE_METADATA_INDEX_COLUMN_STATS.key(),
"true");
+
conf.setString(HoodieMetadataConfig.ENABLE_METADATA_INDEX_PARTITION_STATS.key(),
"false");
Review Comment:
Added a partition-level regression in 04cabae59bcb verifying that
`computeCandidatePartitions()` retains the matching partition with missing name
stats and prunes the fully indexed, nonmatching partition. All 33 targeted
tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
