rangareddy commented on issue #16709: URL: https://github.com/apache/hudi/issues/16709#issuecomment-4900646815
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: on the current master / 1.x line, clustering already appears able to expand the number of output file groups: - The per-group output count is size-driven - `PartitionAwareClusteringPlanStrategy.getNumberOfOutputFileGroups()` returns `ceil(groupSize / hoodie.clustering.plan.strategy.target.file.max.bytes)` (PartitionAwareClusteringPlanStrategy.java:268), so a clustering group can produce more output file groups than it has input file slices. - #18172 (commit cc3a5293b) additionally added the `fileSliceGroup.getRight() > 1` condition in `buildClusteringGroupsForPartition` (PartitionAwareClusteringPlanStrategy.java:113), so even a single large input file slice is now scheduled and split into multiple output file groups - previously such single-input groups were skipped unless `hoodie.clustering.plan.strategy.single.group.clustering.enabled` was set. This change is on master only (not in 0.14.0 / 1.0.0 / 1.0.2 / branch-0.x). Note: #18172 was filed against #17918, not this ticket, so we could not definitively confirm it covers the full intended scope of HUDI-8453. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
