Hi, I'd like to move a discussion from Google Docs to the mailing list so that it's visible to everyone.
*Yuanfeng Hu* brought up two concerns: 1) Related to the resource-stabilization-timeout,he thinks 10s May be too short. In a container environment, if the number of tm added by rest requests is greater than 1, the tm initialization time may be much longer than 10s. and 2) He proposed a little scenario: There is 1 slot in the entire cluster. At this time, my task is running at 1 parallelism (the required slot is also 1). Then I add a tm(1slot), which will obviously trigger a change event, and it will become stable after 10 seconds. If I change the required resources to 3 through rest at this time, rescale will be triggered immediately. and runs at a parallelism of 2, Is this the expected result, or do we expect that the Rescale will be triggered after adding another tm, because this exactly matches the required resources Thank you, *Yuanfeng Hu, *for opening the discussion. --------------------------------------------------------------------------------------- 1) Regarding the stabilization period: I am unsure what you mean by the part, 'if the number of tm added by rest requests is greater than 1.' However, I understand that it can take some time to spawn additional containers/pods in a containerized environment. On the other hand, if a user adds more TMs, for instance, by increasing the number of replicas in a Kubernetes deployment, these replicas should appear with some delay but at a similar time, correct? It's worth mentioning that since FLIP-461 <https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing+for+the+AdaptiveScheduler>, the rescale operation is synchronized with checkpoint events, so the rescale doesn't happen right after this timeout expires. If we believe it is necessary to have different values for the stabilization period in the Executing and WaitingForResources states, even though this increases configuration complexity slightly, we could have separate parameters for these two states: jobmanager.adaptive-scheduler.resource-stabilization-timeout <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-adaptive-scheduler-resource-stabilization-timeout> and *jobmanager.adaptive-scheduler.scaling-stabilization-timeout *(replacing the jobmanager.adaptive-scheduler.scaling-interval.max <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-adaptive-scheduler-scaling-interval-max> ). *2) *Regarding the proposed scenario: The same behavior occurs in the current Flink version when the `min-parallelism-increase` is set to its default value 1. In this case, the rescale operation is triggered immediately or aligned with the checkpoint event (specified in FLIP-461). So, I would say the behavior is expected. Additionally, users can configure the rescaling behavior. For example, if a user sets the lower bound parallelism to 2 and the upper bound to 3, the system will rescale after 10 seconds. Alternatively, if the user sets the same value for the lower and upper bounds, the rescale operation will wait until all slots are available. Best Regrads, Zdenek Tison On Thu, Jul 11, 2024 at 2:38 PM Zdenek Tison <zti...@confluent.io> wrote: > Hello, > > Our team has been working on several improvements for AdaptiveScheduler, > specifically focusing on aligning logic and timeouts in the > WaitingForResources and Executing states. We believe these enhancements > will improve the adaptive scheduler's robustness and maintainability. > > For more detailed information, please refer to the FLIP document. > > https://docs.google.com/document/d/1YeYSs64LqgUr3xyBTCjiRE-CT5VEyHjGjqxnxKPIQhM/edit?usp=sharing > > Thanks, > Zdenek Tison >
