Kaustubh Butte created FLINK-40405:
--------------------------------------
Summary: [FLINK-autoscaler] Add reason tags to
`autoscaler.balanced` counter to distinguish genuinely balanced from blocked
scaling
Key: FLINK-40405
URL: https://issues.apache.org/jira/browse/FLINK-40405
Project: Flink
Issue Type: Improvement
Components: Autoscaler
Affects Versions: 1.20.1, 2.0.0
Reporter: Kaustubh Butte
Currently, the `autoscaler.balanced` counter in `AutoscalerFlinkMetrics` is
incremented every time `ScalingExecutor.scaleResource()` returns `false`.
However, there are at least 7 distinct reasons why scaling might not happen,
and all of them are collapsed into a single counter:
1. *Genuinely balanced* - all vertices are within utilization bounds
(`ScalingExecutor.java`, `!anyVertexOutsideBound`)
2. *Memory pressure* - GC pressure or heap usage exceeds threshold
(`ScalingExecutor.isJobUnderMemoryPressure()`)
3. *Scaling disabled by config* - `SCALING_ENABLED=false`
(`ScalingExecutor.checkIfBlockedAndTriggerScalingEvent()`)
4. *Excluded period* - calendar blackout window active
(`CalendarUtils.inExcludedPeriods()`)
5. *Resource quota exceeded* - CPU/memory quota or cluster capacity reached
(`ScalingExecutor.scalingWouldExceedMaxResources()`)
6. *Ineffective scaling detected* - last scale-up did not improve throughput
(`JobVertexScaler.detectIneffectiveScaleUp()`)
7. *Scale-down cooldown* - `SCALE_DOWN_INTERVAL` has not elapsed
(`JobVertexScaler.applyScaleDownInterval()`)
8. *Data unavailable* – TPR or target rate is NaN
(`JobVertexScaler.computeScaleTargetParallelism()`)
This makes the `balanced` metric unusable for monitoring and alerting. A job
showing `balanced=100, scalings=0` could be perfectly healthy OR completely
blocked by memory pressure, and there is no way to tell from the metric alone.
*Proposed change:*
1. Introduce a `ScaleResult` enum with values for each blocking reason
2. Change `ScalingExecutor.scaleResource()` to return `ScaleResult` instead of
`boolean`
3. In `AutoscalerFlinkMetrics`, register tagged sub-counters using
`MetricGroup.addGroup("reason", reasonTag).counter("balanced")`
4. Keep the existing `numBalanced` counter incrementing for backward
compatibility
5. Enrich `ParallelismChange.noChange()` with a `NoChangeReason` to propagate
vertex-level blocking reasons
*Result in metric reporters:*
- `autoscaler.balanced` (total, backward compatible)
- `autoscaler.balanced.reason.genuinely_balanced`
- `autoscaler.balanced.reason.memory_pressure`
- `autoscaler.balanced.reason.config_disabled`
- `autoscaler.balanced.reason.excluded_period`
- `autoscaler.balanced.reason.resource_quota`
- `autoscaler.balanced.reason.ineffective_scaling`
- `autoscaler.balanced.reason.cooldown`
- `autoscaler.balanced.reason.data_unavailable`
*Files affected:*
- `AutoscalerFlinkMetrics.java` – add tagged counters
- `ScalingExecutor.java` – return `ScaleResult` instead of `boolean`
- `JobAutoScalerImpl.java` – switch on `ScaleResult`
- `JobVertexScaler.java` – add `NoChangeReason` to `ParallelismChange`
- New: `ScaleResult.java`
All changes are in the `flink-autoscaler` module (standalone,
platform-agnostic). Works with both K8s and YARN deployments.
*Why this matters:*
The Flink autoscaler is increasingly used in large-scale production deployments
with thousands of jobs. Operators need to distinguish "this job does not need
scaling" from "this job needs scaling but is blocked" to build effective
dashboards and alerts. Today, the only way to get this information is through
K8s events (which are not available in non-K8s deployments and are not suitable
for metric-based monitoring).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
