Thanks for the tip Ryan.  With the following settings on the table, I
see the metadata folder staying much tidier for my test:

write.metadata.previous-versions-max=20
write.metadata.delete-after-commit.enabled=true
commit.manifest.min-count-to-merge=20
commit.manifest-merge.enabled=true

Cheers,
Peter

> Peter, Iceberg keeps track of the last few metadata files and cleans them
> up after it reaches a configurable max, which I think is defaulted to 100.
>
> On Tue, Jun 15, 2021 at 5:58 PM Peter Giles <gil...@uw.edu> wrote:
>
>> I'm exploring the use of Iceberg (0.11.1) in a spark app, and I'm very
>> enthusiastic about the features, but I am seeing something unexpected in my
>> testing: there is an ever increasing accumulation of v#.metadata.json files
>> for my test table, and I am not seeing them get cleaned up when using any
>> of these procedures:
>>
>> expire_snapshots
>> rewrite_manifests
>> remove_orphan_files
>>
>> Is this expected?  Is there a process for pruning this set of files to
>> just what is needed to support the remaining snapshots?
>>
>> Thank you,
>> Peter
>>
>
>
> --
> Ryan Blue
>

Reply via email to