Vivek,
No, the number of snapshots that are retained is not controlled by this
setting. Snapshots are typically expired when they reach a certain age and
we have a setting to keep some minimum number of snapshots. Here’s the API:
table.expireSnapshots()
.expireOlderThan(oneDayAgoMs)
.retainLast(3)
.commit()
One thing that we’ve considered is keeping these settings in table
properties, so you could just run expireSnapshots().commit() and get the
table’s settings. We haven’t added that yet, though.
Ryan
On Tue, Jun 15, 2021 at 9:52 PM Vivekanand Vellanki <vi...@dremio.com>
wrote:
> Ryan, Does this also mean that the metadata.json files will only contain
> information about the last 100 snapshots, thus preventing the metadata.json
> files from becoming very large?
>
> On Wed, Jun 16, 2021 at 6:30 AM Ryan Blue <b...@apache.org> wrote:
>
>> Peter, Iceberg keeps track of the last few metadata files and cleans them
>> up after it reaches a configurable max, which I think is defaulted to 100.
>>
>> On Tue, Jun 15, 2021 at 5:58 PM Peter Giles <gil...@uw.edu> wrote:
>>
>>> I'm exploring the use of Iceberg (0.11.1) in a spark app, and I'm very
>>> enthusiastic about the features, but I am seeing something unexpected in my
>>> testing: there is an ever increasing accumulation of v#.metadata.json files
>>> for my test table, and I am not seeing them get cleaned up when using any
>>> of these procedures:
>>>
>>> expire_snapshots
>>> rewrite_manifests
>>> remove_orphan_files
>>>
>>> Is this expected? Is there a process for pruning this set of files to
>>> just what is needed to support the remaining snapshots?
>>>
>>> Thank you,
>>> Peter
>>>
>>
>>
>> --
>> Ryan Blue
>>
>
--
Ryan Blue
