[
https://issues.apache.org/jira/browse/FLINK-36780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17901005#comment-17901005
]
Mark Cho commented on FLINK-36780:
----------------------------------
This seems to be a bug in the Kafka Source connector as the
[documentation|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#dynamic-partition-discovery]
states that this behavior is enabled by default. However, it turns out it's
unintentionally disabled by default without additional config.
> Kafka source disable partition discovery unexpectedly
> -----------------------------------------------------
>
> Key: FLINK-36780
> URL: https://issues.apache.org/jira/browse/FLINK-36780
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: 1.18.1, 1.20.0, 1.19.1, 2.0-preview
> Reporter: Mingliang Liu
> Priority: Major
> Labels: pull-request-available
>
> Currently Kafka source enables partition discovery. This is set by
> {{partition.discovery.interval.ms}}, aka
> {{KafkaSourceOptions.PARTITION_DISCOVERY_INTERVAL_MS}}. The default value is
> 5 minutes, which is equal to the default value of {{metadata.max.age.ms}} in
> Kafka.
> However, it's disabled by default unexpectedly in the source builder
> ([code|https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L476-L480]).
> The intention I believe was to only disable for bounded source.
> We need a fix that is able to keep the default partition discovery. This
> could cause data loss after Kafka retention if the new partitions are not
> consumed silently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
