Hi everyone, I would like to start a discussion on FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source[1].
As described in mail thread[2], dynamic partition discovery is disabled by default and users have to explicitly specify the interval of discovery in order to turn it on. Besides, if the initial offset strategy is LATEST, same strategy is used for new partitions, leading to the loss of some data (thinking a new partition is created and might be discovered by Kafka source several minutes later, and the message produced into the partition within the gap might be dropped if we use for example "latest" as the initial offset strategy.) The goals of this FLIP are as follows: 1. Enable partition discovery by default. 2. Use earliest as the offset strategy for new partitions after the first discovery. Looking forward to hearing from you. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source [2] <https://lists.apache.org/thread/d7zy46gj3sw0zwzq2rj3fmc0hx8ojtln> https://lists.apache.org/thread/d7zy46gj3sw0zwzq2rj3fmc0hx8ojtln Best, Hongshun
