Thanks for the tip Ryan. With the following settings on the table, I see the metadata folder staying much tidier for my test:
write.metadata.previous-versions-max=20 write.metadata.delete-after-commit.enabled=true commit.manifest.min-count-to-merge=20 commit.manifest-merge.enabled=true Cheers, Peter > Peter, Iceberg keeps track of the last few metadata files and cleans them > up after it reaches a configurable max, which I think is defaulted to 100. > > On Tue, Jun 15, 2021 at 5:58 PM Peter Giles <gil...@uw.edu> wrote: > >> I'm exploring the use of Iceberg (0.11.1) in a spark app, and I'm very >> enthusiastic about the features, but I am seeing something unexpected in my >> testing: there is an ever increasing accumulation of v#.metadata.json files >> for my test table, and I am not seeing them get cleaned up when using any >> of these procedures: >> >> expire_snapshots >> rewrite_manifests >> remove_orphan_files >> >> Is this expected? Is there a process for pruning this set of files to >> just what is needed to support the remaining snapshots? >> >> Thank you, >> Peter >> > > > -- > Ryan Blue >