Re: Flink custom source

Hi Ahmed, Thanks again. You are right. send a split request call in the reader fixed the issue. Thanks On Sun, Oct 6, 2024 at 11:30 AM Ahmed Hamdy wrote: > Hi Anil, > Yes you are right, split assignment is either proactive by split discovery > and then assignment as in the scheduled discover

Re: Flink custom source

Hi Anil, Yes you are right, split assignment is either proactive by split discovery and then assignment as in the scheduled discovery tasks in Kafka, JDBC sources or reactive where the reader itself submits a split request and the enumerator assigns on requests like the example you added, or even a

Re: Flink custom source

Hi Ahmed, Thanks for the response. This is the part that I find unclear in the documentation and FLIP-27. The actual split assignment happens in the SplitEnumerator#handleSplitRequest method and not in SplitEnumerator#start. Both KafkaSource and JdbcSource use context.callAsync to identify new spli

Re: Flink custom source

Hi Anil I am glad the new (yet to be only) source API is getting attention, according to the execution model the assignments of splits is the responsibility of the enumerator and it seems that the enumerator is not assigning the readers any splits. Check kafka source for reference[1] 1- https://git