Re: Flink table maintenance

2024-04-21 Thread Gen Luo
al processing time is limited, since the inbound queue will be >>full, or the mentioned AsyncIO issue will bite us. >> >> Step 3 should happen only once, and fast - we do not need AsyncIO here >> >> So I think we should use AsyncIO in the maintenance tasks where it is >&g

Re: Flink table maintenance

2024-04-18 Thread Gen Luo
> and we need to provide enough resources to this application, so we are able > to finish the maintenance tasks. When there are no maintenance tasks, then > this application doesn't do anything (essentially idling the resources). I > don't see yet, how this is different than s

Re: Flink table maintenance

2024-04-18 Thread Gen Luo
rote: > Hi Gen, > > Thanks for your interest and thoughts! > See my answers and questions below: > > Gen Luo ezt írta (időpont: 2024. ápr. 17., Sze, > 20:32): > >> Hi, >> >> Sorry for joining the discussion so late. I'm from the flink community >> and

Re: Flink table maintenance

2024-04-17 Thread Gen Luo
Hi, Sorry for joining the discussion so late. I'm from the flink community and did some work about the SinkV2. I'd like to share some thoughts from the view of flink. While I'm quite new to Iceberg, please feel free to correct me if I'm making any mistakes. 1. checkpointing I think the maintenanc