Hi, In my experience with the librdkafka client and the Go wrapper, the topic-pattern subscribe is reactive. The Flink Kafka connector might behave similarly.
Best, Denis On Fri, Oct 15, 2021 at 12:34 AM Preston Price <nacro...@gmail.com> wrote: > No, the topic-pattern won't work for my case. Topics that I should > subscribe to can be enabled/disabled based on settings I read from another > system, so there's no way to craft a single regular expression that would > fit the state of all potential topics. Additionally the documentation you > linked seems to suggest that the regular expression is evaluated only once > "when the job starts running". My understanding is it would not pick up new > topics that match the pattern after the job starts. > > > On Wed, Oct 13, 2021 at 8:51 PM Caizhi Weng <tsreape...@gmail.com> wrote: > >> Hi! >> >> I suppose you want to read from different topics every now and then? Does >> the topic-pattern option [1] in Table API Kafka connector meet your needs? >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/kafka/#topic-pattern >> >> Preston Price <nacro...@gmail.com> 于2021年10月14日周四 上午1:34写道： >> >>> The KafkaSource, and KafkaSourceBuilder appear to prevent users from >>> providing their own KafkaSubscriber. Am I overlooking something? >>> >>> In my case I have an external system that controls which topics we >>> should be ingesting, and it can change over time. I need to add, and remove >>> topics as we refresh configuration from this external system without having >>> to stop and start our Flink job. Initially it appeared I could accomplish >>> this by providing my own implementation of the `KafkaSubscriber` interface, >>> which would be invoked periodically as configured by the ` >>> partition.discovery.interval.ms` property. However there is no way to >>> provide my implementation to the KafkaSource since the constructor for >>> KafkaSource is package protected, and the KafkaSourceBuilder does not >>> supply a way to provide the `KafkaSubscriber`. >>> >>> How can I accomplish a period refresh of the topics to ingest? >>> >>> Thanks >>> >>> >>> -- Regards, Denis Nutiu
