Hi Preston,

if you still need to set KafkaSubscriber explicitly, could you please
create a feature request for that? For now, you probably have to resort to
reflection hacks and build against a the non-public KafkaSubscriber.

On Fri, Oct 15, 2021 at 4:03 AM Prasanna kumar <
prasannakumarram...@gmail.com> wrote:

> Yes you are right.
>
> We tested recently to find that the flink jobs do not pick up the new
> topics that got created with the same pattern provided to flink kafka
> consumer.  The topics are set only during the start of the jobs.
>
> Prasanna.
>
> On Fri, 15 Oct 2021, 05:44 Preston Price, <nacro...@gmail.com> wrote:
>
>> Okay so topic discovery is possible with topic patterns, and maybe topic
>> lists. However I don't believe it's possible to change the configured topic
>> list, or topic pattern after the source is created.
>>
>> On Thu, Oct 14, 2021, 3:52 PM Denis Nutiu <denis.nu...@gmail.com> wrote:
>>
>>> There is a setting for dynamic topic discovery
>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kafka/#topic-and-partition-discovery
>>>
>>> Best,
>>>
>>> Denis
>>>
>>> On Fri, Oct 15, 2021 at 12:48 AM Denis Nutiu <denis.nu...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> In my experience with the librdkafka client and the Go wrapper, the
>>>> topic-pattern subscribe is reactive. The Flink Kafka connector might behave
>>>> similarly.
>>>>
>>>> Best,
>>>> Denis
>>>>
>>>> On Fri, Oct 15, 2021 at 12:34 AM Preston Price <nacro...@gmail.com>
>>>> wrote:
>>>>
>>>>> No, the topic-pattern won't work for my case. Topics that I should
>>>>> subscribe to can be enabled/disabled based on settings I read from another
>>>>> system, so there's no way to craft a single regular expression that would
>>>>> fit the state of all potential topics. Additionally the documentation you
>>>>> linked seems to suggest that the regular expression is evaluated only once
>>>>> "when the job starts running". My understanding is it would not pick up 
>>>>> new
>>>>> topics that match the pattern after the job starts.
>>>>>
>>>>>
>>>>> On Wed, Oct 13, 2021 at 8:51 PM Caizhi Weng <tsreape...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I suppose you want to read from different topics every now and then?
>>>>>> Does the topic-pattern option [1] in Table API Kafka connector meet your
>>>>>> needs?
>>>>>>
>>>>>> [1]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/kafka/#topic-pattern
>>>>>>
>>>>>> Preston Price <nacro...@gmail.com> 于2021年10月14日周四 上午1:34写道:
>>>>>>
>>>>>>> The KafkaSource, and KafkaSourceBuilder appear to prevent users from
>>>>>>> providing their own KafkaSubscriber. Am I overlooking something?
>>>>>>>
>>>>>>> In my case I have an external system that controls which topics we
>>>>>>> should be ingesting, and it can change over time. I need to add, and 
>>>>>>> remove
>>>>>>> topics as we refresh configuration from this external system without 
>>>>>>> having
>>>>>>> to stop and start our Flink job. Initially it appeared I could 
>>>>>>> accomplish
>>>>>>> this by providing my own implementation of the `KafkaSubscriber` 
>>>>>>> interface,
>>>>>>> which would be invoked periodically as configured by the `
>>>>>>> partition.discovery.interval.ms` property. However there is no way
>>>>>>> to provide my implementation to the KafkaSource since the constructor 
>>>>>>> for
>>>>>>> KafkaSource is package protected, and the KafkaSourceBuilder does not
>>>>>>> supply a way to provide the `KafkaSubscriber`.
>>>>>>>
>>>>>>> How can I accomplish a period refresh of the topics to ingest?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Denis Nutiu
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Denis Nutiu
>>>
>>

Reply via email to