Hi Mason, Thank you for driving this proposal!
Currently, Autoscaler only supports the maximum source parallelism of KafkaSource. Introducing the generic metric to support it is good to me, +1 for this proposal. I have a question: You added the metric in the flink repo, and Autoscaler will fetch this metric. But I didn't see any connector to register this metric. Currently, only IteratorSourceEnumerator setUnassignedSplitsGauge, and KafkaSource didn't register it. IIUC, if we don't do it, autoscaler still cannot fetch this metric, right? If yes, I suggest this FLIP includes registering metric part, otherwise these metrics still cannot work. Please correct me if I misunderstood anything, thanks~ Best, Rui On Fri, Nov 17, 2023 at 6:53 AM Mason Chen <mas.chen6...@gmail.com> wrote: > Hi all, > > I would like to start a discussion on FLIP-394: Add Metrics for Connector > Agnostic Autoscaling [1]. > > This FLIP recommends adding two metrics to make autoscaling work for > bounded split source implementations like IcebergSource. These metrics are > required by the Flink Kubernetes Operator autoscaler algorithm [2] to > retrieve information for the backlog and the maximum source parallelism. > The changes would affect the `@PublicEvolving` `SplitEnumeratorMetricGroup` > API of the source connector framework. > > Best, > Mason > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-394%3A+Add+Metrics+for+Connector+Agnostic+Autoscaling > [2] > > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/#limitations >
