Yes you are right. We tested recently to find that the flink jobs do not pick up the new topics that got created with the same pattern provided to flink kafka consumer. The topics are set only during the start of the jobs.
Prasanna. On Fri, 15 Oct 2021, 05:44 Preston Price, <nacro...@gmail.com> wrote: > Okay so topic discovery is possible with topic patterns, and maybe topic > lists. However I don't believe it's possible to change the configured topic > list, or topic pattern after the source is created. > > On Thu, Oct 14, 2021, 3:52 PM Denis Nutiu <denis.nu...@gmail.com> wrote: > >> There is a setting for dynamic topic discovery >> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kafka/#topic-and-partition-discovery >> >> Best, >> >> Denis >> >> On Fri, Oct 15, 2021 at 12:48 AM Denis Nutiu <denis.nu...@gmail.com> >> wrote: >> >>> Hi, >>> >>> In my experience with the librdkafka client and the Go wrapper, the >>> topic-pattern subscribe is reactive. The Flink Kafka connector might behave >>> similarly. >>> >>> Best, >>> Denis >>> >>> On Fri, Oct 15, 2021 at 12:34 AM Preston Price <nacro...@gmail.com> >>> wrote: >>> >>>> No, the topic-pattern won't work for my case. Topics that I should >>>> subscribe to can be enabled/disabled based on settings I read from another >>>> system, so there's no way to craft a single regular expression that would >>>> fit the state of all potential topics. Additionally the documentation you >>>> linked seems to suggest that the regular expression is evaluated only once >>>> "when the job starts running". My understanding is it would not pick up new >>>> topics that match the pattern after the job starts. >>>> >>>> >>>> On Wed, Oct 13, 2021 at 8:51 PM Caizhi Weng <tsreape...@gmail.com> >>>> wrote: >>>> >>>>> Hi! >>>>> >>>>> I suppose you want to read from different topics every now and then? >>>>> Does the topic-pattern option [1] in Table API Kafka connector meet your >>>>> needs? >>>>> >>>>> [1] >>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/kafka/#topic-pattern >>>>> >>>>> Preston Price <nacro...@gmail.com> 于2021年10月14日周四 上午1:34写道： >>>>> >>>>>> The KafkaSource, and KafkaSourceBuilder appear to prevent users from >>>>>> providing their own KafkaSubscriber. Am I overlooking something? >>>>>> >>>>>> In my case I have an external system that controls which topics we >>>>>> should be ingesting, and it can change over time. I need to add, and >>>>>> remove >>>>>> topics as we refresh configuration from this external system without >>>>>> having >>>>>> to stop and start our Flink job. Initially it appeared I could accomplish >>>>>> this by providing my own implementation of the `KafkaSubscriber` >>>>>> interface, >>>>>> which would be invoked periodically as configured by the ` >>>>>> partition.discovery.interval.ms` property. However there is no way >>>>>> to provide my implementation to the KafkaSource since the constructor for >>>>>> KafkaSource is package protected, and the KafkaSourceBuilder does not >>>>>> supply a way to provide the `KafkaSubscriber`. >>>>>> >>>>>> How can I accomplish a period refresh of the topics to ingest? >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> >>> >>> -- >>> Regards, >>> Denis Nutiu >>> >> >> >> -- >> Regards, >> Denis Nutiu >> >
