Hi,Raman Verma*:* Generally, Kafka responds quickly. However, as an asynchronous operation, we cannot guarantee that there will be no abnormal operations, such as temporary network issues. The two cases I mentioned are special situations to better understand the concept.
Best Hongshun On Fri, Apr 28, 2023 at 9:48 AM Raman Verma <raman.x.ve...@gmail.com> wrote: > Hello Hongshun Wang, > > You have mentioned that the first partition discovery can be very slow > (section: Why do we need initialDiscoveryFinished?) > > Do you mean that Kafka can be slow to respond. If so, any idea under what > conditions Kafka would be slow. > Or, is it just a matter of bad timing, where this call does not return > before checkpoint. > > Thanks, > Raman Verma > > On 2023/03/17 10:41:40 Hongshun Wang wrote: > > Hi everyone, > > > > I would like to start a discussion on FLIP-288:Enable Dynamic Partition > > Discovery by Default in Kafka Source[1]. > > > > As described in mail thread[2], dynamic partition discovery is disabled > by > > default and users have to explicitly specify the interval of discovery in > > order to turn it on. Besides, if the initial offset strategy is LATEST, > > same strategy is used for new partitions, leading to the loss of some > data > > (thinking a new partition is created and might be discovered by Kafka > > source several minutes later, and the message produced into the partition > > within the gap might be dropped if we use for example "latest" as the > > initial offset strategy.) > > > > The goals of this FLIP are as follows: > > > > 1. Enable partition discovery by default. > > 2. Use earliest as the offset strategy for new partitions after the > > first discovery. > > > > Looking forward to hearing from you. > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source > > > > [2] <https://lists.apache.org/thread/d7zy46gj3sw0zwzq2rj3fmc0hx8ojtln> > > https://lists.apache.org/thread/d7zy46gj3sw0zwzq2rj3fmc0hx8ojtln > > > > > > Best, > > > > Hongshun > > > > > Sent from my iPad
