Hi Preston, if you still need to set KafkaSubscriber explicitly, could you please create a feature request for that? For now, you probably have to resort to reflection hacks and build against a the non-public KafkaSubscriber.
On Fri, Oct 15, 2021 at 4:03 AM Prasanna kumar < prasannakumarram...@gmail.com> wrote: > Yes you are right. > > We tested recently to find that the flink jobs do not pick up the new > topics that got created with the same pattern provided to flink kafka > consumer. The topics are set only during the start of the jobs. > > Prasanna. > > On Fri, 15 Oct 2021, 05:44 Preston Price, <nacro...@gmail.com> wrote: > >> Okay so topic discovery is possible with topic patterns, and maybe topic >> lists. However I don't believe it's possible to change the configured topic >> list, or topic pattern after the source is created. >> >> On Thu, Oct 14, 2021, 3:52 PM Denis Nutiu <denis.nu...@gmail.com> wrote: >> >>> There is a setting for dynamic topic discovery >>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kafka/#topic-and-partition-discovery >>> >>> Best, >>> >>> Denis >>> >>> On Fri, Oct 15, 2021 at 12:48 AM Denis Nutiu <denis.nu...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> In my experience with the librdkafka client and the Go wrapper, the >>>> topic-pattern subscribe is reactive. The Flink Kafka connector might behave >>>> similarly. >>>> >>>> Best, >>>> Denis >>>> >>>> On Fri, Oct 15, 2021 at 12:34 AM Preston Price <nacro...@gmail.com> >>>> wrote: >>>> >>>>> No, the topic-pattern won't work for my case. Topics that I should >>>>> subscribe to can be enabled/disabled based on settings I read from another >>>>> system, so there's no way to craft a single regular expression that would >>>>> fit the state of all potential topics. Additionally the documentation you >>>>> linked seems to suggest that the regular expression is evaluated only once >>>>> "when the job starts running". My understanding is it would not pick up >>>>> new >>>>> topics that match the pattern after the job starts. >>>>> >>>>> >>>>> On Wed, Oct 13, 2021 at 8:51 PM Caizhi Weng <tsreape...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi! >>>>>> >>>>>> I suppose you want to read from different topics every now and then? >>>>>> Does the topic-pattern option [1] in Table API Kafka connector meet your >>>>>> needs? >>>>>> >>>>>> [1] >>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/kafka/#topic-pattern >>>>>> >>>>>> Preston Price <nacro...@gmail.com> 于2021年10月14日周四 上午1:34写道： >>>>>> >>>>>>> The KafkaSource, and KafkaSourceBuilder appear to prevent users from >>>>>>> providing their own KafkaSubscriber. Am I overlooking something? >>>>>>> >>>>>>> In my case I have an external system that controls which topics we >>>>>>> should be ingesting, and it can change over time. I need to add, and >>>>>>> remove >>>>>>> topics as we refresh configuration from this external system without >>>>>>> having >>>>>>> to stop and start our Flink job. Initially it appeared I could >>>>>>> accomplish >>>>>>> this by providing my own implementation of the `KafkaSubscriber` >>>>>>> interface, >>>>>>> which would be invoked periodically as configured by the ` >>>>>>> partition.discovery.interval.ms` property. However there is no way >>>>>>> to provide my implementation to the KafkaSource since the constructor >>>>>>> for >>>>>>> KafkaSource is package protected, and the KafkaSourceBuilder does not >>>>>>> supply a way to provide the `KafkaSubscriber`. >>>>>>> >>>>>>> How can I accomplish a period refresh of the topics to ingest? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> >>>> >>>> -- >>>> Regards, >>>> Denis Nutiu >>>> >>> >>> >>> -- >>> Regards, >>> Denis Nutiu >>> >>
