+1 for supporting more maintenance support in Flink Peter, just wondering if there is really any known opposition/dissenting opinions or if you're just looking for general agreement on the path forward?
I would also agree with the single pipeline / post commit approach as having to configure multiple jobs or scheduling is a lot of additional infrastructure work to set up, so single feels like it provides the most immediate value for the larger community. -Dan On Tue, May 7, 2024 at 6:32 AM Zhu Zhu <reed...@gmail.com> wrote: > +1 > > Thanks, > Zhu > > Jean-Baptiste Onofré <j...@nanthrax.net> 于2024年5月7日周二 16:17写道： > >> +1 >> >> Regards >> JB >> >> On Fri, May 3, 2024 at 8:30 PM Péter Váry <peter.vary.apa...@gmail.com> >> wrote: >> > >> > Hi everyone, >> > >> > I would like to make a proposal [1] to support Flink Table Maintenance >> in Iceberg. The main goal is to have a solution where Flink can execute the >> Maintenance Tasks as part of the streaming job. Especially Rewrite Data >> Files, Rewrite Manifest Files and Expire Snapshots. >> > The secondary goal is to provide building blocks for Flink batch jobs >> to execute the Maintenance Tasks independently, where the scheduling is >> done outside of Flink. >> > >> > This proposal is the outcome of extensive community discussions on the >> mailing list [2, 3]. >> > >> > Please respond with your recommendation: >> > +1 if you support moving forward with the two separate objects model. >> > 0 if you are neutral. >> > -1 if you disagree with the two separate objects model. >> > >> > Thanks, >> > Peter >> > >> > [1] https://github.com/apache/iceberg/issues/10264 >> > [2] https://lists.apache.org/thread/yjcwbf1037jdq4prty6rtrrqmjzc71o0 >> > [3] https://lists.apache.org/thread/10mdf9zo6pn0dfq791nf4w1m7jh9k3sl >> >
