Re: missing files in an Iceberg table

These people using S3 versioned buckets? If so, until actually purged, they are just hiding under tombstone markers Our little cloud-storage support-call library, cloudstore, has something to list and recover these https://github.com/steveloughran/cloudstore https://github.com/steveloughran/clou

Re: missing files in an Iceberg table

Dan, Thanks for the pointers. Let me look into that work. - Wing Yew On Tue, Jan 28, 2025 at 8:49 AM Daniel Weeks wrote: > Hey Wing Yew, > > I would agree that this is a common problem and we need a way to get > tables back into a good state when something unexpected happens. Amogh and > Matt

Re: missing files in an Iceberg table

Hey Wing Yew, I would agree that this is a common problem and we need a way to get tables back into a good state when something unexpected happens. Amogh and Matt have a PR (API: Define RepairManifests action interface #10784) that was originally

Re: missing files in an Iceberg table

Hi Wing, Thank you for bringing this up. We run into this all the time, particularly when the underlying storage has data management settings outside of Iceberg's ownership (I.E. s3 retention policies). It is probably a weekly occurrence, and one of the biggest pain points for new builders. Thanks

Re: missing files in an Iceberg table

Hi, I can also confirm that there are a number of users who find themselves unintentionally deleting some files and not being able to use their Iceberg tables anymore. The number of these incidents is surprisingly high for some reason. There was also a question on Iceberg Slack around this problem