Hi Mason, thanks for creating that.
We are happy to take contribuitons (I flagged it as a starter task). On Wed, Oct 27, 2021 at 2:36 AM Mason Chen <mason.c...@apple.com> wrote: > Hi all, > > I have a similar requirement to Preston. I created > https://issues.apache.org/jira/browse/FLINK-24660 to track this effort. > > Best, > Mason > > On Oct 18, 2021, at 1:59 AM, Arvid Heise <ar...@apache.org> wrote: > > Hi Preston, > > if you still need to set KafkaSubscriber explicitly, could you please > create a feature request for that? For now, you probably have to resort to > reflection hacks and build against a the non-public KafkaSubscriber. > > On Fri, Oct 15, 2021 at 4:03 AM Prasanna kumar < > prasannakumarram...@gmail.com> wrote: > >> Yes you are right. >> >> We tested recently to find that the flink jobs do not pick up the new >> topics that got created with the same pattern provided to flink kafka >> consumer. The topics are set only during the start of the jobs. >> >> Prasanna. >> >> On Fri, 15 Oct 2021, 05:44 Preston Price, <nacro...@gmail.com> wrote: >> >>> Okay so topic discovery is possible with topic patterns, and maybe topic >>> lists. However I don't believe it's possible to change the configured topic >>> list, or topic pattern after the source is created. >>> >>> On Thu, Oct 14, 2021, 3:52 PM Denis Nutiu <denis.nu...@gmail.com> wrote: >>> >>>> There is a setting for dynamic topic discovery >>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kafka/#topic-and-partition-discovery >>>> >>>> Best, >>>> >>>> Denis >>>> >>>> On Fri, Oct 15, 2021 at 12:48 AM Denis Nutiu <denis.nu...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> In my experience with the librdkafka client and the Go wrapper, the >>>>> topic-pattern subscribe is reactive. The Flink Kafka connector might >>>>> behave >>>>> similarly. >>>>> >>>>> Best, >>>>> Denis >>>>> >>>>> On Fri, Oct 15, 2021 at 12:34 AM Preston Price <nacro...@gmail.com> >>>>> wrote: >>>>> >>>>>> No, the topic-pattern won't work for my case. Topics that I should >>>>>> subscribe to can be enabled/disabled based on settings I read from >>>>>> another >>>>>> system, so there's no way to craft a single regular expression that would >>>>>> fit the state of all potential topics. Additionally the documentation you >>>>>> linked seems to suggest that the regular expression is evaluated only >>>>>> once >>>>>> "when the job starts running". My understanding is it would not pick up >>>>>> new >>>>>> topics that match the pattern after the job starts. >>>>>> >>>>>> >>>>>> On Wed, Oct 13, 2021 at 8:51 PM Caizhi Weng <tsreape...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> I suppose you want to read from different topics every now and then? >>>>>>> Does the topic-pattern option [1] in Table API Kafka connector meet your >>>>>>> needs? >>>>>>> >>>>>>> [1] >>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/kafka/#topic-pattern >>>>>>> >>>>>>> Preston Price <nacro...@gmail.com> 于2021年10月14日周四 上午1:34写道： >>>>>>> >>>>>>>> The KafkaSource, and KafkaSourceBuilder appear to prevent users >>>>>>>> from providing their own KafkaSubscriber. Am I overlooking something? >>>>>>>> >>>>>>>> In my case I have an external system that controls which topics we >>>>>>>> should be ingesting, and it can change over time. I need to add, and >>>>>>>> remove >>>>>>>> topics as we refresh configuration from this external system without >>>>>>>> having >>>>>>>> to stop and start our Flink job. Initially it appeared I could >>>>>>>> accomplish >>>>>>>> this by providing my own implementation of the `KafkaSubscriber` >>>>>>>> interface, >>>>>>>> which would be invoked periodically as configured by the ` >>>>>>>> partition.discovery.interval.ms` property. However there is no way >>>>>>>> to provide my implementation to the KafkaSource since the constructor >>>>>>>> for >>>>>>>> KafkaSource is package protected, and the KafkaSourceBuilder does not >>>>>>>> supply a way to provide the `KafkaSubscriber`. >>>>>>>> >>>>>>>> How can I accomplish a period refresh of the topics to ingest? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Denis Nutiu >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Denis Nutiu >>>> >>> >
