yihua opened a new pull request, #20267: URL: https://github.com/apache/hudi/pull/20267
### Describe the issue this Pull Request addresses closes #20266 part of #20257 On a METADATA_ONLY bootstrapped table with column stats enabled (the Spark default for later writes), the column stats of a bootstrapped file group come from its skeleton file, which only has the meta columns. Partition stats pruning then drops bootstrapped partitions, and column stats pruning drops skeleton files when the filter also references a meta column, so data-column filters with data skipping on return too few rows. Partition stats pruning came from #10352 and #11556 (1.0.0), neither of which considered bootstrap skeleton files. Filling null stats for a file missing a column came from #5746; #10389 made that safe for columns unindexed across the whole table, which does not cover a single file missing an indexed column. Column stats became on by default for Spark writers in #12596 (1.0.1), which made this reachable with default settings. ### Summary and Changelog Column stats and partition stats pruning no longer drop bootstrapped data they have no data-column stats for. - `PartitionStatsIndexSupport.prunePartitions` does not prune partitions when the table has a bootstrap base path. - `ColumnStatsIndexSupport.computeCandidateFileNames` keeps base files that have a bootstrap base file (skeleton files) as candidates, only on tables with a bootstrap base path. - Test: `TestDataSourceForBootstrap#testMetadataBootstrapDataSkippingWithColumnStats` (COW and MOR, partition stats on and off) bootstraps, enables column stats on later writes, and checks that seven filters return the same rows with data skipping on and off. All four cases fail on master (e.g. `driver >= 'driver_5'` returns 18 rows instead of 56). The companion bootstrap fixes (column stats initialization failing after a METADATA_ONLY bootstrap, and the record index on bootstrap tables) follow in separate PRs; the column stats initialization one should land after this, since it makes this path reachable right after the bootstrap. This PR touches `ColumnStatsIndexSupport` next to #20096, in different hunks. ### Impact Fixes silent wrong results for data skipping on bootstrapped tables. Bootstrapped tables (including FULL_RECORD ones) no longer use partition stats for pruning, and skeleton files are never skipped by column stats; tables without a bootstrap base path are unchanged and do no extra work. Indexing the bootstrap source files' stats would restore that pruning and can follow separately. Known remaining gap, not a regression: the expression index on a bootstrapped table is also built from the skeleton file (`SparkMetadataWriterUtils` reads `new HoodieBaseFile(filePath)` without its bootstrap source), and with partition stats pruning now skipped such tables reach expression index partition pruning in `HoodieFileIndex`; this is not addressed here. Data skipping for `hoodie.bootstrap.data.queries.only` reads is handled by a separate fix. ### Risk Level low: the change only removes pruning on tables with a bootstrap base path, so it can read more files there but cannot drop rows. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
