Re: Iceberg table maintenance

2024-05-03 Thread Péter Váry
Thanks everyone for participating in the discussion! This was really helpful! I have updated the document, and hopefully incorporated all of the comments, suggestions, and finalized the framework part of the document. Left the part describing the specific Maintenance Tasks in draft (marked them in

Re: Iceberg table maintenance

2024-04-22 Thread Péter Váry
Hi Nathan Ma, Thanks for joining the discussion! 1-2. In the other thread ( https://lists.apache.org/thread/10mdf9zo6pn0dfq791nf4w1m7jh9k3sl) we discussed the resource consumption/buffers/checkpointing in detail. We agreed that even if we try to do everything to separate the stream processing fro

Re: Iceberg table maintenance

2024-04-22 Thread nathan ma
Hi guys, thanks for raising this wonderful thread, and thanks to @ peter.vary.apa...@gmail.com, the doc really makes things clear I have several questions to catch up and I did't find answers in the document. I would appreciate that anyone could give some feedbacks cause the issues really importa

Re: Iceberg table maintenance

2024-04-22 Thread Péter Váry
Hi Team, The discussion continued on the previous thread by Gen Luo and Zhu Zhu. See: https://lists.apache.org/thread/10mdf9zo6pn0dfq791nf4w1m7jh9k3sl. Adding them to this thread too, so we can continue the discussion in one place. Based on their thoughts there, the current suggestion is: Q: Do y

Re: Iceberg table maintenance

2024-04-11 Thread Brian Olsen
Hey everyone, Following up from this meeting, we decided due to the distributed nature of everyone involved on Flink, it would be best to achieve consensus on some critical points that Péter outlined for us. - Do you have concerns with the Maintenance Tasks execution infrastructure? The cu

RE: Iceberg table maintenance

2024-04-11 Thread wenjin
Hi Peter, I am interested in your proposal and have clarified some confusions with your help in Flink community, thanks for your answers. I participated in yesterday’s discussion, but since I am not a native English speaker, I am concerned that I may have missed some details regarding the foll

Re: Iceberg table maintenance

2024-04-08 Thread Péter Váry
Sry. This was meant for the Flink dev list :( On Tue, Apr 9, 2024, 06:48 Péter Váry wrote: > Forwarding the invite for the discussion we plan to do with the Iceberg > folks, as some of you might be interested in this. > > -- Forwarded message - > From: Brian Olsen > Date: Mon, A