Re: [Proposal] REST Spec: Server-side Metadata Tables

2024-07-06 Thread Walaa Eldin Moustafa
Hi Yufie, The original proposal did not seem to indicate that the metadata tables will be "materialized" (outside regular Iceberg metadata since most of those metadata tables are actually "views" on Iceberg metadata). However, in the last response, it seems metadata could potentially be written to

Re: [DISCUSS] Extend Snapshot Metadata Lifecycle

2024-07-06 Thread Walaa Eldin Moustafa
Hi Szehon, Thanks for sharing this proposal. We have thought along the same lines and implemented an external system (LakeChime [1]) that retains snapshot + partition metadata for longer (actual internal implementation keeps data for 13 months, but that can be tuned). For efficient analysis, we ha