Hello Hongshun Wang, You have mentioned that the first partition discovery can be very slow (section: Why do we need initialDiscoveryFinished?)
Do you mean that Kafka can be slow to respond. If so, any idea under what conditions Kafka would be slow. Or, is it just a matter of bad timing, where this call does not return before checkpoint. Thanks, Raman Verma On 2023/03/17 10:41:40 Hongshun Wang wrote: > Hi everyone, > > I would like to start a discussion on FLIP-288:Enable Dynamic Partition > Discovery by Default in Kafka Source[1]. > > As described in mail thread[2], dynamic partition discovery is disabled by > default and users have to explicitly specify the interval of discovery in > order to turn it on. Besides, if the initial offset strategy is LATEST, > same strategy is used for new partitions, leading to the loss of some data > (thinking a new partition is created and might be discovered by Kafka > source several minutes later, and the message produced into the partition > within the gap might be dropped if we use for example "latest" as the > initial offset strategy.) > > The goals of this FLIP are as follows: > > 1. Enable partition discovery by default. > 2. Use earliest as the offset strategy for new partitions after the > first discovery. > > Looking forward to hearing from you. > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source > > [2] <https://lists.apache.org/thread/d7zy46gj3sw0zwzq2rj3fmc0hx8ojtln> > https://lists.apache.org/thread/d7zy46gj3sw0zwzq2rj3fmc0hx8ojtln > > > Best, > > Hongshun > Sent from my iPad
