kaustubhbutte17 opened a new pull request, #1218:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1218
## What is the purpose of the change
The autoscaler reports three job level counters: `autoscaler.scalings`,
`autoscaler.errors` and `autoscaler.balanced`. The `balanced` counter
increments
whenever an evaluation cycle applies no parallelism change. Eight different
causes share that single counter:
- the job is genuinely at its target parallelism
- `job.autoscaler.scaling.enabled` is false
- the current time is inside `job.autoscaler.excluded.periods`
- GC pressure or heap usage is above the limit
- the change exceeds the CPU or memory quota
- the cluster cannot schedule the TaskManagers that the change needs
- a custom scaling executor vetoed the change
- a vertex is blocked by an ineffective previous scale up, by the scale down
interval, or by a missing processing rate
A job that reports `balanced=100, scalings=0` is therefore either perfectly
healthy or completely blocked. An operator cannot tell which from the metric,
and must read the logs of the job instead. This makes a fleet wide rollout of
autoscaling hard to supervise.
This change makes each outcome visible as its own counter.
## Brief change log
- Add `ScaleResult`, an enum with one constant per outcome and a stable lower
case tag.
- `ScalingExecutor.scaleResource` and `ScalingExecutor.execute` return
`ScaleResult` in place of `boolean`. Each path that applies no change
returns
its own constant.
- `AutoscalerFlinkMetrics.incrementBalanced(ScaleResult)` increments the
existing untagged `balanced` counter, then increments a counter under the
`reason` metric variable.
- Add `ParallelismChange.NoChangeReason` so that vertex level causes, which
are
the ineffective scale up, the scale down cooldown and the missing data
paths,
reach the job level.
- Split the resource limit check into its cluster capacity branch and its
quota
branch, because an operator resolves the two differently.
- Update the metric and the internals documentation, in English and Chinese.
## Design notes
**Backward compatibility.** The untagged `autoscaler.balanced` counter keeps
its
meaning and its total. Existing dashboards and alerts need no change. The
tagged
counters are additive.
**Not a public API break.** `ScalingExecutor.execute` has one caller,
`JobAutoScalerImpl.runScalingLogic`, and `JobAutoScaler` carries `@Internal`.
**Metric cardinality.** A tagged counter is created the first time the
autoscaler reports that reason. A job therefore carries counters only for the
reasons it really hits, which is usually two or three, and not all ten.
**`NoChangeReason` stays out of `equals` and `hashCode`.**
`JobVertexScalerTest`
compares against `ParallelismChange.noChange(n)` in more than 60 places. The
reason explains a decision and does not define it, which follows the
treatment
that `outsideUtilizationBound` already gets in the no-change case.
`testNoChangeReasonIsExcludedFromEquality` guards this.
**Cooldown reporting.** A delayed scale down reports the cooldown only when
the
vertex is outside the utilization bound. A vertex inside the bound is
dropped by
the balanced gate anyway, so a cooldown tag would wrongly suggest that the
parallelism drops when the interval ends.
**Two constants go beyond the JIRA description.** The JIRA predates both
paths.
- `BLOCKED_BY_CUSTOM_EXECUTOR` covers the `ScalingExecutorPlugin` veto.
- `BLOCKED_BY_CLUSTER_RESOURCES` separates the cluster capacity check from
the
quota check.
**Diff size.** Most of the diff is the test migration. About 33 assertions in
`ScalingExecutorTest` change from `assertTrue` or `assertFalse` to an
assertion
on the exact `ScaleResult`. That is churn, not new logic, and each one now
states which reason the path reports.
## Verifying this change
`mvn -pl flink-autoscaler test` passes, with 306 tests.
New tests:
- `AutoScalerFlinkMetricsTest.testBalancedCounterIsTaggedWithTheReason`
checks
that the untagged counter holds the total and the tagged counters split it.
-
`AutoScalerFlinkMetricsTest.testTaggedCounterIsRegisteredOnlyForReportedReasons`
checks the lazy registration.
- `JobVertexScalerTest.testNoChangeReasonIsExcludedFromEquality`
-
`JobVertexScalerTest.testNoChangeReasonIsDataUnavailableWhenProcessingRateIsNaN`
-
`JobVertexScalerTest.testNoChangeReasonIsCooldownOnlyForAnOutOfBoundScaleDown`
Existing tests in `ScalingExecutorTest` now assert the exact reason, which
covers the config disabled, excluded period, memory pressure, cluster
resource,
quota and custom executor veto paths.
Each new test was confirmed to fail when the behaviour it covers is broken on
purpose, and to pass again after the code is restored.
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- Core observability classes: **yes**, `AutoscalerFlinkMetrics` gains
counters.
Existing counters are unchanged.
- The autoscaler decision logic: **no**, every scaling decision is identical.
Only the reporting of the decision changes.
## Documentation
- Does this pull request introduce a new feature? **yes**, per reason
counters
on `autoscaler.balanced`.
- If yes, how is the feature documented? **docs**, in
`docs/content/docs/operations/metrics.md` and
`docs/content/docs/internals/autoscaler.md`, with the Chinese mirrors
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
