Ryan, Does this also mean that the metadata.json files will only contain information about the last 100 snapshots, thus preventing the metadata.json files from becoming very large?
On Wed, Jun 16, 2021 at 6:30 AM Ryan Blue <b...@apache.org> wrote: > Peter, Iceberg keeps track of the last few metadata files and cleans them > up after it reaches a configurable max, which I think is defaulted to 100. > > On Tue, Jun 15, 2021 at 5:58 PM Peter Giles <gil...@uw.edu> wrote: > >> I'm exploring the use of Iceberg (0.11.1) in a spark app, and I'm very >> enthusiastic about the features, but I am seeing something unexpected in my >> testing: there is an ever increasing accumulation of v#.metadata.json files >> for my test table, and I am not seeing them get cleaned up when using any >> of these procedures: >> >> expire_snapshots >> rewrite_manifests >> remove_orphan_files >> >> Is this expected? Is there a process for pruning this set of files to >> just what is needed to support the remaining snapshots? >> >> Thank you, >> Peter >> > > > -- > Ryan Blue >
