[
https://issues.apache.org/jira/browse/FLINK-40405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18122183#comment-18122183
]
Kaustubh Butte commented on FLINK-40405:
----------------------------------------
I opened a pull request for this ticket:
https://github.com/apache/flink-kubernetes-operator/pull/1218
The change adds a ScaleResult enum, with one constant for each outcome of an
evaluation cycle. AutoscalerFlinkMetrics.incrementBalanced increments the
untagged autoscaler.balanced counter, then a counter below the "reason" metric
variable. The untagged counter keeps its total, so existing dashboards continue
to operate.
The change is in the flink-autoscaler module. A standalone or YARN deployment
that uses the library also gets the tags.
Two constants are not in the description above. BLOCKED_BY_CUSTOM_EXECUTOR
covers the ScalingExecutorPlugin veto, which is newer than this ticket.
BLOCKED_BY_CLUSTER_RESOURCES separates the cluster capacity check from the
quota check, because an operator corrects the two differently.
Please assign the ticket to me.
> [FLINK-autoscaler] Add reason tags to `autoscaler.balanced` counter to
> distinguish genuinely balanced from blocked scaling
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40405
> URL: https://issues.apache.org/jira/browse/FLINK-40405
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Affects Versions: 2.0.0, 1.20.1
> Reporter: Kaustubh Butte
> Priority: Major
> Labels: autoscaling, observability, pull-request-available
>
> Currently, the `autoscaler.balanced` counter in `AutoscalerFlinkMetrics` is
> incremented every time `ScalingExecutor.scaleResource()` returns `false`.
> However, there are at least 7 distinct reasons why scaling might not happen,
> and all of them are collapsed into a single counter:
> 1. *Genuinely balanced* - all vertices are within utilization bounds
> (`ScalingExecutor.java`, `!anyVertexOutsideBound`)
> 2. *Memory pressure* - GC pressure or heap usage exceeds threshold
> (`ScalingExecutor.isJobUnderMemoryPressure()`)
> 3. *Scaling disabled by config* - `SCALING_ENABLED=false`
> (`ScalingExecutor.checkIfBlockedAndTriggerScalingEvent()`)
> 4. *Excluded period* - calendar blackout window active
> (`CalendarUtils.inExcludedPeriods()`)
> 5. *Resource quota exceeded* - CPU/memory quota or cluster capacity reached
> (`ScalingExecutor.scalingWouldExceedMaxResources()`)
> 6. *Ineffective scaling detected* - last scale-up did not improve throughput
> (`JobVertexScaler.detectIneffectiveScaleUp()`)
> 7. *Scale-down cooldown* - `SCALE_DOWN_INTERVAL` has not elapsed
> (`JobVertexScaler.applyScaleDownInterval()`)
> 8. *Data unavailable* – TPR or target rate is NaN
> (`JobVertexScaler.computeScaleTargetParallelism()`)
> This makes the `balanced` metric unusable for monitoring and alerting. A job
> showing `balanced=100, scalings=0` could be perfectly healthy OR completely
> blocked by memory pressure, and there is no way to tell from the metric alone.
> *Proposed change:*
> 1. Introduce a `ScaleResult` enum with values for each blocking reason
> 2. Change `ScalingExecutor.scaleResource()` to return `ScaleResult` instead
> of `boolean`
> 3. In `AutoscalerFlinkMetrics`, register tagged sub-counters using
> `MetricGroup.addGroup("reason", reasonTag).counter("balanced")`
> 4. Keep the existing `numBalanced` counter incrementing for backward
> compatibility
> 5. Enrich `ParallelismChange.noChange()` with a `NoChangeReason` to propagate
> vertex-level blocking reasons
> *Result in metric reporters:*
> - `autoscaler.balanced` (total, backward compatible)
> - `autoscaler.balanced.reason.genuinely_balanced`
> - `autoscaler.balanced.reason.memory_pressure`
> - `autoscaler.balanced.reason.config_disabled`
> - `autoscaler.balanced.reason.excluded_period`
> - `autoscaler.balanced.reason.resource_quota`
> - `autoscaler.balanced.reason.ineffective_scaling`
> - `autoscaler.balanced.reason.cooldown`
> - `autoscaler.balanced.reason.data_unavailable`
> *Files affected:*
> - `AutoscalerFlinkMetrics.java` – add tagged counters
> - `ScalingExecutor.java` – return `ScaleResult` instead of `boolean`
> - `JobAutoScalerImpl.java` – switch on `ScaleResult`
> - `JobVertexScaler.java` – add `NoChangeReason` to `ParallelismChange`
> - New: `ScaleResult.java`
> All changes are in the `flink-autoscaler` module (standalone,
> platform-agnostic). Works with both K8s and YARN deployments.
> *Why this matters:*
> The Flink autoscaler is increasingly used in large-scale production
> deployments with thousands of jobs. Operators need to distinguish "this job
> does not need scaling" from "this job needs scaling but is blocked" to build
> effective dashboards and alerts. Today, the only way to get this information
> is through K8s events (which are not available in non-K8s deployments and are
> not suitable for metric-based monitoring).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
