Re: Ad-hoc partition bucketing

2023-07-06 Thread Joseph Allemandou
Thank you Manu and Russell for your answers, Would there be any document / ticket / commit where I could find some information or example on how the partition transforms are implemented, and the various code places they involve touching ? Many thanks again :) Joseph On Wed, Jul 5, 2023 at 1:43 P

Re: Ad-hoc partition bucketing

2023-07-05 Thread russell . spitzer
We have been discussing something like this as well, either an arbitrary partitioning scheme or just a more extensive and customizable transform. An example I’m interested in is a geo hash index where we store offsets on a large grid to denote partitions. The total offset file for the whole plan

Re: Ad-hoc partition bucketing

2023-07-04 Thread Manu Zhang
Hi Joseph, You may find everything about partitioning and bucketing under https://iceberg.apache.org/spec/#partition-transforms. I don't think we can add new bucketing functions now. I'm also curious whether we can have bucketing functions at table definition such that partitioning will be consist