Hi all,
rewrite_data_files selects files by size and delete counts. A table whose
files are all size-healthy but overlap on the sort key is never selected
for sort/zorder compaction: every run reports success and clustering never
improves. There is also no metric that detects this state. Issue #17489 has
a deterministic two-table reproduction — identical data, the size-healthy
copy reports rewritten=0 at overlap depth 8, the fragmented control
rewrites to depth 1.
PR #17504 <https://github.com/apache/iceberg/pull/17504> implements both
halves:
compute_sort_order_stats: a read-only Spark procedure reporting
per-partition overlap depth, computed from manifest lower/upper bounds only
(follows the compute_partition_stats pattern).
min-overlap-depth: an opt-in sort strategy option that rewrites the files
behind that depth. This part follows Peter Vary's suggestion on the PR to
compute the regions where overlap exceeds a threshold and mark those files.
Defaults are unchanged - the option is unset by default and planning reads
bounds only for the sort column, only when set.
CI is green. Peter suggested finding a committer with interest in this
area, hence this mail.
On review load: the PR is three commits, and commit 1 is a self-contained
283-line core handler (sweep line over bounds, plus tests) - reviewing that
alone validates or kills the approach. Commits 2-3 are thin layers on top
(procedure, planner option). I'm also happy to split it into separate PRs
if that is preferred.
Related work: this composes with the K-way merge strategy proposed in PR
#16305 (detect by overlap, rewrite without shuffle) - I've left review
notes there on how the two could line up.
Any feedback on the direction, naming, or output shape is welcome.
Thanks, Heekyung (gimgit)
