Hi Neelesh,
Thank you for organizing the meeting earlier today and starting this thread. I think it’s awesome to see the community align not just on types but implementations for Variant.
I wanted to highlight one additional item I heard discussed on the sync. - We need to align on an inference algorithm that defines how a variant object is shredded.
- The goal is we can align using data from high quality benchmarking data sets if anyone or company is willing to contribute.
Best Regards, Kurtis C. Wright
Hi all, Variant shredding writers now exist across iceberg-java, arrow-go, and arrow-rs, and shredded write paths are landing soon in iceberg-go, iceberg-rust, and (future) iceberg-python. We don't yet have a shared policy on whether to shred, how the layout is declared, and how it's expressed so every implementation produces compatible output. I'd like to propose one: Variant Shredding Policy Across Iceberg Implementations. Please take a look. Next steps:
1. The community reviews the proposal and agrees on the shredding representation.
2. Once consensus is reached, I'll start a [VOTE] thread.
3. After the vote passes, I'll land the adjustments in the Java implementation.
4. iceberg-go, iceberg-rust, and others follow to align.
Thanks,
Neelesh
|
