[
https://issues.apache.org/jira/browse/FLINK-32020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743029#comment-17743029
]
Qingsheng Ren commented on FLINK-32020:
---------------------------------------
Merged to main: 811716c5155e82fa3bfc47ced53daef53bb99cce
> Enable Dynamic Partition Discovery by Default in Kafka Source based on
> FLIP-288
> -------------------------------------------------------------------------------
>
> Key: FLINK-32020
> URL: https://issues.apache.org/jira/browse/FLINK-32020
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka
> Affects Versions: kafka-3.0.0
> Reporter: Hongshun Wang
> Assignee: Hongshun Wang
> Priority: Major
> Labels: pull-request-available
> Fix For: kafka-4.0.0
>
>
> As described in
> [FLIP-288|https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source],
> dynamic partition discovery is disabled by default, and users have to
> specify the interval of discovery in order to turn it on.
> This subtask is to enable Dynamic Partition Discovery by Default in Kafka
> Source.
> Partition discovery is performed on the KafkaSourceEnumerator, which
> asynchronously fetches topic metadata from the Kafka cluster and checks if
> there are any new topics and partitions. This should not cause performance
> issues on the Flink side.
> On the Kafka broker side, partition discovery sends a MetadataRequest to the
> Kafka broker to fetch topic information. Considering that the Kafka broker
> has its metadata cache and the default request frequency is relatively low
> (once every 30 seconds), this is not a heavy operation, and the broker's
> performance will not be significantly affected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
