Hi all, Just following up on this thread. The subject header may have been misleading--I was proposing to make KafkaSubscriber @PublicEvolving and expose a setter to pass a custom implementation. It seems logical since the KafkaSource is also @PublicEvolving and this lets the user know that the interface may be subject to change if requirements change in the future. Anyone have any opinions? Thanks!
Best, Mason On Wed, Apr 13, 2022 at 10:07 AM Mason Chen <mas.chen6...@gmail.com> wrote: > Hi Chesnay, > > Typically, users want to plug in a KafkaSubscriber that depends on an > external system [1][2]. We could also provide a higher level interface that > doesn’t depend on the Kafka Admin Client, but I think it would be more > flexible to be able to re-use the one created by the enumerator if needed. > If we don't want to expose the Kafka Admin Client and if users want to > apply some complex filter, then we can also provide a pluggable interface > used in a similar implementation to that of the subscriber used for topic > pattern and allow users to filter topics after the Kafka API response. > > [1] https://www.mail-archive.com/user@flink.apache.org/msg44340.html > [2] https://www.mail-archive.com/dev@flink.apache.org/msg52007.html > > Best, > Mason > > > On Wed, Apr 13, 2022 at 6:32 AM Chesnay Schepler <ches...@apache.org> > wrote: > >> Could you expand a bit on possible alternative implementations that >> require this interface to become public, opposed to providing more >> built-in ways to subscribe? >> >> On 13/04/2022 11:26, Qingsheng Ren wrote: >> > Thanks for the proposal Mason! I think exposing `KafkaSubscriber` as >> public API is helpful for users to implement more complex subscription >> logics. >> > >> > +1 (non-binding) >> > >> > Cheers, >> > >> > Qingsheng >> > >> >> On Apr 12, 2022, at 11:46, Mason Chen <mas.chen6...@gmail.com> wrote: >> >> >> >> Hi Flink Devs, >> >> >> >> I was looking to contribute to >> https://issues.apache.org/jira/browse/FLINK-24660, which is a ticket to >> track changing the KafkaSubscriber from Internal to PublicEvolving. >> >> >> >> In the PR, it seems a few of us have agreement on making the >> subscriber pluggable in the KafkaSource, but I'd like to raise the question >> nevertheless. Furthermore, there is also interest from various Flink >> mailing threads and on the Jira ticket itself for the ticket, so I think >> the change would be beneficial to the users. There is already some feedback >> to make the contract of handling removed splits by the KafkaSource and >> subscriber clearer in the docs. >> >> >> >> I have yet to address all the PR feedback, but does anyone have any >> concerns before I proceed further? >> >> >> >> Best, >> >> Mason >> >> >>
