[
https://issues.apache.org/jira/browse/FLINK-40405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaustubh Butte updated FLINK-40405:
-----------------------------------
Labels: autoscaling observability (was: )
> [FLINK-autoscaler] Add reason tags to `autoscaler.balanced` counter to
> distinguish genuinely balanced from blocked scaling
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40405
> URL: https://issues.apache.org/jira/browse/FLINK-40405
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Affects Versions: 2.0.0, 1.20.1
> Reporter: Kaustubh Butte
> Priority: Major
> Labels: autoscaling, observability
>
> Currently, the `autoscaler.balanced` counter in `AutoscalerFlinkMetrics` is
> incremented every time `ScalingExecutor.scaleResource()` returns `false`.
> However, there are at least 7 distinct reasons why scaling might not happen,
> and all of them are collapsed into a single counter:
> 1. *Genuinely balanced* - all vertices are within utilization bounds
> (`ScalingExecutor.java`, `!anyVertexOutsideBound`)
> 2. *Memory pressure* - GC pressure or heap usage exceeds threshold
> (`ScalingExecutor.isJobUnderMemoryPressure()`)
> 3. *Scaling disabled by config* - `SCALING_ENABLED=false`
> (`ScalingExecutor.checkIfBlockedAndTriggerScalingEvent()`)
> 4. *Excluded period* - calendar blackout window active
> (`CalendarUtils.inExcludedPeriods()`)
> 5. *Resource quota exceeded* - CPU/memory quota or cluster capacity reached
> (`ScalingExecutor.scalingWouldExceedMaxResources()`)
> 6. *Ineffective scaling detected* - last scale-up did not improve throughput
> (`JobVertexScaler.detectIneffectiveScaleUp()`)
> 7. *Scale-down cooldown* - `SCALE_DOWN_INTERVAL` has not elapsed
> (`JobVertexScaler.applyScaleDownInterval()`)
> 8. *Data unavailable* – TPR or target rate is NaN
> (`JobVertexScaler.computeScaleTargetParallelism()`)
> This makes the `balanced` metric unusable for monitoring and alerting. A job
> showing `balanced=100, scalings=0` could be perfectly healthy OR completely
> blocked by memory pressure, and there is no way to tell from the metric alone.
> *Proposed change:*
> 1. Introduce a `ScaleResult` enum with values for each blocking reason
> 2. Change `ScalingExecutor.scaleResource()` to return `ScaleResult` instead
> of `boolean`
> 3. In `AutoscalerFlinkMetrics`, register tagged sub-counters using
> `MetricGroup.addGroup("reason", reasonTag).counter("balanced")`
> 4. Keep the existing `numBalanced` counter incrementing for backward
> compatibility
> 5. Enrich `ParallelismChange.noChange()` with a `NoChangeReason` to propagate
> vertex-level blocking reasons
> *Result in metric reporters:*
> - `autoscaler.balanced` (total, backward compatible)
> - `autoscaler.balanced.reason.genuinely_balanced`
> - `autoscaler.balanced.reason.memory_pressure`
> - `autoscaler.balanced.reason.config_disabled`
> - `autoscaler.balanced.reason.excluded_period`
> - `autoscaler.balanced.reason.resource_quota`
> - `autoscaler.balanced.reason.ineffective_scaling`
> - `autoscaler.balanced.reason.cooldown`
> - `autoscaler.balanced.reason.data_unavailable`
> *Files affected:*
> - `AutoscalerFlinkMetrics.java` – add tagged counters
> - `ScalingExecutor.java` – return `ScaleResult` instead of `boolean`
> - `JobAutoScalerImpl.java` – switch on `ScaleResult`
> - `JobVertexScaler.java` – add `NoChangeReason` to `ParallelismChange`
> - New: `ScaleResult.java`
> All changes are in the `flink-autoscaler` module (standalone,
> platform-agnostic). Works with both K8s and YARN deployments.
> *Why this matters:*
> The Flink autoscaler is increasingly used in large-scale production
> deployments with thousands of jobs. Operators need to distinguish "this job
> does not need scaling" from "this job needs scaling but is blocked" to build
> effective dashboards and alerts. Today, the only way to get this information
> is through K8s events (which are not available in non-K8s deployments and are
> not suitable for metric-based monitoring).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
