[
https://issues.apache.org/jira/browse/FLINK-40455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyula Fora reassigned FLINK-40455:
----------------------------------
Assignee: Dennis-Mircea Ciupitu
> Decoupling Autotuning from the Autoscaling logic
> ------------------------------------------------
>
> Key: FLINK-40455
> URL: https://issues.apache.org/jira/browse/FLINK-40455
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler, Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Assignee: Dennis-Mircea Ciupitu
> Priority: Major
>
> h1. Background
> Autotuning is currently applied only as part of autoscaling decisions, so new
> tuning recommendations are computed only when the autoscaler decides to
> rescale, and not at all when the autoscaler is disabled or doesn't take a
> scaling decision. This in practice limits the following concrete scenarios:
> * Jobs that keep parallelism fixed by design cannot benefit from any tuning.
> * Jobs that were scaled down to the lowest parallelism possible cannot be
> further tuned such that the resource allocation can be optimized down even
> more over long periods of low traffic.
> * Jobs that were scaled up to the highest parallelism possible cannot be
> further tuned such that the resource allocation can be optimized up even more
> over long periods of low traffic.
> With the above limitations, autotuning logic is unable to further realize
> cost and infrastructure optimizations. Therefore, those limitations bring the
> need to have the autotuning decoupled from autoscaling logic, with
> coordination in between.
> As part of the [dev mailing list
> discussion|https://lists.apache.org/thread/vk6j8793rx2y78vhh1yvfjf4qntpfkm9]
> we agreed to take the direction of decoupling autotuning from autoscaling
> logic within the autoscaler module.
> h1. Goal
> The main purpose of this task is to cover the following:
> * Autotuning model is refactored to a more extensible, self-contained model.
> * Autotuning logic is applied either within the autoscaling logic or as a
> separate step after autoscaling, depending on the case, under a full
> coordination mechanism.
> * Autoscaler metric collection will be shared by both autoscaling and the
> autotuning.
> * Autotuning is getting its own trigger interval, cooldown, and
> minimum-improvement guard.
> By implementing the above refactorings, autotuning will have a stronger
> baseline on top of which multiple autotuners can be built. This will also
> open the door to further enhancements and capabilities, making it an even
> more self-contained, stable, and reliable feature, capable of delivering real
> cost and infrastructure optimizations.
> h1. Out of scope
> The following items/improvements will stay on hold until the autotuning gets
> further adoption and gets stabilized enough:
> * Move the autotuner into its own module.
> * Store the autotuning's `configOverrides` state into a dedicated autotuner
> ConfigMap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
