Hey guys,
I am an aspiring committer and am super interested in this change. I use Iceberg at reasonably high levels of concurrency in my pipelines, and would love to just be a fly on the wall in discussions (hopefully more). Best, AK On Fri, 30 May 2025 at 6:51 PM, Ryan Blue <rdb...@gmail.com> wrote: > does it make sense to take metadata json file into consideration as well? > Currently it is just a large json string containing all snapshots. Since it > is also on the critical path of a commit, I’m not sure if we can explore > incremental semantics on it together with manifest list files to reduce the > commit overhead. > > No, table-level metadata and the content metadata tree are separate > concepts. We’ve addressed table-level metadata and the snapshot list by > introducing change-based commits in the REST protocol. This work aims to > make updates to the content metadata tree more lightweight. > > On Thu, May 29, 2025 at 6:37 PM Gang Wu <ust...@gmail.com> wrote: > >> This is a long-awaited discussion! >> >> BTW, does it make sense to take metadata json file into consideration as >> well? Currently it is just a large json string containing all snapshots. >> Since it is also on the critical path of a commit, I'm not sure if we can >> explore incremental semantics on it together with manifest list files to >> reduce the commit overhead. >> >> Best, >> Gang >> >> On Fri, May 30, 2025 at 7:10 AM Steven Wu <stevenz...@gmail.com> wrote: >> >>> This will be great for users. metadata can self adapt. Start with a >>> compacted one file. As the table grows in size, the metadata can adapt to a >>> tree or linked structure. >>> >>> On Thu, May 29, 2025 at 3:44 PM Russell Spitzer < >>> russell.spit...@gmail.com> wrote: >>> >>>> I’m also super excited about this idea >>>> >>>> On Thu, May 29, 2025 at 3:37 PM Amogh Jahagirdar <2am...@gmail.com> >>>> wrote: >>>> >>>>> Thanks for kicking this thread off Ryan, I'm interested in helping out >>>>> here! I've been working on a proposal in this area and it would be great >>>>> to >>>>> collaborate with different folks and exchange ideas here, since I think a >>>>> lot of people are interested in solving this problem. >>>>> >>>>> Thanks, >>>>> Amogh Jahagirdar >>>>> >>>>> On Thu, May 29, 2025 at 2:25 PM Ryan Blue <rdb...@gmail.com> wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> Like Russell’s recent note, I’m starting a thread to connect those of >>>>>> us that are interested in the idea of changing Iceberg’s metadata in v4 >>>>>> so >>>>>> that in most cases committing a change only requires writing one >>>>>> additional >>>>>> metadata file. >>>>>> >>>>>> *Idea: One-file commits* >>>>>> >>>>>> The current Iceberg metadata structure requires writing at least one >>>>>> manifest and a new manifest list to produce a new snapshot. The goal of >>>>>> this work is to allow more flexibility by allowing the manifest list >>>>>> layer >>>>>> to store data and delete files. As a result, only one file write would be >>>>>> needed before committing the new snapshot. In addition, this work will >>>>>> also >>>>>> try to explore: >>>>>> >>>>>> - Avoiding small manifests that must be read in parallel and >>>>>> later compacted (metadata maintenance changes) >>>>>> - Extend metadata skipping to use aggregated column ranges that >>>>>> are compatible with geospatial data (manifest metadata) >>>>>> - Using soft deletes to avoid rewriting existing manifests >>>>>> (metadata DVs) >>>>>> >>>>>> If you’re interested in these problems, please reply! >>>>>> >>>>>> Ryan >>>>>> >>>>>
