[ https://issues.apache.org/jira/browse/FLINK-36535 ]
Rui Fan deleted comment on FLINK-36535:
---------------------------------
was (Author: fanrui):
Thanks [~heigebupahei] for the comment!
{quote}that we need to treat job.autoscaler.scale-down.interval as a window.
The default is last value of window, but in some cases it is max in
window（>24hour)?
{quote}
For solution2, yes.
For solution1, the default is last value of window. And users could set an
option to choose the max as the result.
{quote}But it seems to me like we just need to change the default behavior to
max in window?
Just imagine, if you follow the default 1hour configuration, the result of
using max in window is not much different from before, so I think we only need
to change the strategy to max of window and abandon the previous last value of
window.
{quote}
It's a good point.
As I understand, you mean:
* We choose the max as the result by default
* Don't need any additional option, and autoscaler doesn't support choose the
latest parallelism as the result for scale down
Is my understanding correct?
If yes, I think it makes sense for most of scenarios. However I'm afraid it
could not meet all requirements. For example:
* when the job owners are very familiar with the daily traffic changes of the
job, they could set a appropriate job.autoscaler.scale-down.interval, and job
will only scale down once a day when autoscaler choose the latest parallelism.
* IIUC, autoscaler needs at least 2 scales down a day if autoscaler choose the
max parallelism in window.
** Because the max is not the latest parallelism, and after scaling down, max
is most likely not the latest appropriate parallelism for vertex. So job needs
scale down again later.
Please correct me if I misunderstand anything, thanks~
> Optimize the scale down logic based on historical parallelism
> -------------------------------------------------------------
>
> Key: FLINK-36535
> URL: https://issues.apache.org/jira/browse/FLINK-36535
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Reporter: Rui Fan
> Assignee: Rui Fan
> Priority: Major
>
> This is a follow-up to FLINK-36018 . FLINK-36018 supported the lazy scale
> down to avoid frequent rescaling.
> h1. Proposed Change
> Treat scale-down.interval as a window:
> * Recording the scale down trigger time when the recommended parallelism <
> current parallelism
> ** When the recommended parallelism >= current parallelism, cancel the
> triggered scale down
> * The scale down will be executed when currentTime - triggerTime >
> scale-down.interval
> ** {color:#de350b}Change1{color}: Using the maximum parallelism within the
> window instead of the latest parallelism when scaling down.
> * {color:#de350b}Change2{color}: Never scale down when currentTime -
> triggerTime < scale-down.interval
> ** In the FLINK-36018, the scale down may be executed when currentTime -
> triggerTime < scale-down.interval.
> ** For example: the taskA may scale down when taskB needs to scale up.
> h1. Background
> Some critical Flink jobs need to scale up in time, but only scale down on a
> daily basis. In other words, Flink users do not want Flink jobs to be scaled
> down multiple times within 24 hours, and the jobs run at the same parallelism
> as during the peak hours of each day.
> Note: Users hope to scale down only happens when the parallelism during peak
> hours is still a waste of resources. This is a trade-off between downtime and
> resource waste for a critical job.
> h1. Current solution
> In general, this requirement could be met after setting{color:#de350b}
> job.autoscaler.scale-down.interval= 24 hour{color}. When taskA runs with 100
> parallelism, and recommended parallelism is 100 during the peak hours of each
> day. We hope taskA doesn't rescale forever, because the triggered scale down
> will be canceled once the recommended parallelism >= current parallelism
> within 24 hours (It‘s exactly what FLINK-36018 does).
> h1. Unexpected Scenario & how to solve?
> But I found the critical production job is still rescaled about 10 times
> every day (when scale-down.interval is set to 24 hours).
> Root cause: There may be many sources in a job, and the traffic peaks of
> these sources may occur at different times. When taskA triggers scale down,
> the scale down of taskA will not be actively executed within 24 hours, but it
> may be executed when other tasks are scaled up.
> For example:
> * The scale down of sourceB and sourceC may be executed when SourceA scales
> up.
> * After a while, the scale down of sourceA and sourceC may be executed when
> SourceB scales up.
> * After a while, the scale down of sourceA and sourceB may be executed when
> SourceC scales up.
> * When there are many tasks, the above 3 steps will be executed repeatedly.
> That's why the job is rescaled about 10 times every day, the
> {color:#de350b}change2{color} of proposed change could solve this issue:
> Never scale down when currentTime - triggerTime < scale-down.interval.
>
> {color:#de350b}Change1{color}: Using the maximum parallelism within the
> window instead of the latest parallelism when scaling down.
> * It can ensure that the parallelism after scaling down is the parallelism
> at yesterday's peak.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
