Hi Gabor I think the question is "when". As it's a behavior change, I don't think we should do that on a "minor" release, else users would be "surprised". I would propose to keep the current behavior on Iceberg Java 1.x and change the flag to true on Iceberg Java 2.x (after a vote).
Regards JB On Fri, Mar 14, 2025 at 12:18 PM Gabor Kaszab <gaborkas...@apache.org> wrote: > > Hi Iceberg Community, > > There were recent additions to RemoveSnapshots to expire the unused partition > specs and schemas. This is controlled by a flag called 'cleanExpiredMetadata' > and has a default value 'false'. Additionally, Spark and Flink don't offer a > way to set this flag currently. > > 1) Default value of RemoveSnapshots.cleanExpiredMetadata > I'm wondering if it's desired by the community to default this flag to true. > The effect of that would be that each snapshot expiration would also clean up > the unused partition specs and schemas too. This functionality is quite new > so this might need some extra confidence by the community before turning on > by default but I think it's worth a consideration. > > 2) Spark and Flink to support setting this flag > I think it makes sense to add support in Spark's ExpireSnapshotProcedure and > ExpireSnapshotsSparkAction also to Flink's ExpireSnapshotsProcessor and > ExpireSnapshots to allow setting this flag based on (user) inputs. > > WDYT? > > Regards, > Gabor
