+1 for the proposal of adding more table maintenance to Flink. It is great that the maintenance actions can be run in two modes: (1) embedded in the Flink writer job as post commit stage (2) standalone Flink batch job/action. I probably wouldn't label the two goals as primary and secondary. Different users may have different preferences. Or different maintenance tasks may favor one mode over another.
On Fri, May 3, 2024 at 11:30 AM Péter Váry <peter.vary.apa...@gmail.com> wrote: > Hi everyone, > > I would like to make a proposal [1] to support Flink Table Maintenance in > Iceberg. The main goal is to have a solution where Flink can execute the > Maintenance Tasks as part of the streaming job. Especially Rewrite Data > Files, Rewrite Manifest Files and Expire Snapshots. > The secondary goal is to provide building blocks for Flink batch jobs to > execute the Maintenance Tasks independently, where the scheduling is done > outside of Flink. > > This proposal is the outcome of extensive community discussions on the > mailing list [2, 3]. > > Please respond with your recommendation: > +1 if you support moving forward with the two separate objects model. > 0 if you are neutral. > -1 if you disagree with the two separate objects model. > > Thanks, > Peter > > [1] https://github.com/apache/iceberg/issues/10264 > [2] https://lists.apache.org/thread/yjcwbf1037jdq4prty6rtrrqmjzc71o0 > [3] https://lists.apache.org/thread/10mdf9zo6pn0dfq791nf4w1m7jh9k3sl >
