[
https://issues.apache.org/jira/browse/FLINK-39981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-39981:
-----------------------------------
Labels: pull-request-available (was: )
> Autoscaler cannot trigger source rebalance after active split removal on
> dynamic Kafka source
> ---------------------------------------------------------------------------------------------
>
> Key: FLINK-39981
> URL: https://issues.apache.org/jira/browse/FLINK-39981
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler, Kubernetes Operator
> Reporter: Xin Gao
> Priority: Critical
> Labels: pull-request-available
>
> The autoscaler derives Kafka source partition count by enumerating metric
> names matching:
> KafkaSourceReader.topic.<topic>.partition.<id>.currentOffset
> After DynamicKafkaSource removes Kafka clusters/topics/splits due to metadata
> changes, removed split metric names may remain visible through the aggregated
> metrics endpoint. The autoscaler therefore continues counting removed
> partitions as active partitions.
> *Example:*
> - Source parallelism: 6
> - Active Kafka partitions before metadata update: 8
> - Metadata update removes a Kafka cluster containing 5 partitions
> - Active Kafka partitions after update: 3
> - Removed partition currentOffset metric names remain visible
> - Autoscaler still calculates numSourcePartitions = 8
> - Source parallelism remains 6
> *Expected behavior:*
> The autoscaler should calculate numSourcePartitions from currently {*}active
> Kafka splits only{*}. After the metadata update, numSourcePartitions should
> become 3 and source parallelism should be capped to <= 3 on the next
> autoscaler evaluation.
> *Actual behavior:*
> The autoscaler continues counting stale currentOffset metric names and does
> not scale down. The source remains {*}over-parallelized with empty
> subtasks{*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
