Hi folks, I’ve opened a PR to add support for Hilbert curve ordering in rewrite_data_files, offering an alternative spatial clustering strategy alongside the existing Z-order implementation.
*CALL system.rewrite_data_files( table => 'db.tbl', strategy => 'sort', sort_order => 'hilbert(c1, c2)' );* When mapping multi-column space onto a 1D sequence for file compaction, the goal is to keep points that are close in space close together on disk to maximize file skipping during range filters.While Z-order works well, it suffers from periodic "jumps" across space. Hilbert curves avoid these discontinuous jumps, neighboring points on the curve are always true neighbors in the multi-dimensional space. In practice, this provides tighter spatial clustering and better data skipping (improving reading performance, reducing the number of parquet files analyzed per query) when queries filter across multiple columns simultaneously. Implementation: - Reuses existing infra: Under the hood, the implementation leverages Iceberg's existing, well-tested Z-order byte encodings and builds the curve math on top. The existing Z-order codebase remains completely untouched. - Algorithm & Context: Based on J. Skilling’s standard algorithm, *"Programming the Hilbert curve"* (AIP Conf. Proc. 707, 381, 2004 <https://doi.org/10.1063/1.1751381>; see also UMD Reference <https://drum.lib.umd.edu/items/c713788f-517c-4c2e-a6a2-7ea9d7e69c5b>). Note that a similar approach has been proven in the ecosystem, with Apache Hudi supporting Hilbert curves since v0.10.0 (details here <https://hudi.apache.org/blog/2021/12/29/hudi-zorder-and-hilbert-space-filling-curves/> ). - Scope: Currently limited to Spark 4.1 (other engines will fall back gracefully). - Testing: Ships with full test coverage including curve-correctness checks (bijection and neighbor-locality guarantees), runner/action/SQL integration tests. No benchmark provided. I’d love to get feedback from the community on the approach and would appreciate a review when you have a moment: https://github.com/apache/iceberg/pull/16827 Looking forward to your thoughts! Cheers, Gianluca
