The AdaptiveScheduler is not limited to reactive mode. There are no
deployment limitations for the scheduler itself.
In a nutshell, all that reactive mode does is crank the target
parallelism to infinity, when usually it is the parallelism the user has
set in the job/configuration.
I think it would be fine if a new/revised rescale API were only
available in the Adaptive Scheduler (without reactive mode!) for starters.
We'd require way more stuff to make this useful for batch workloads.
On 10/10/2022 16:47, Maximilian Michels wrote:
Hey Gyula,
Is the Adaptive Scheduler limited to the Reactive mode? I agree that if we
move forward with the Adaptive Scheduler solution it should support all
deployment scenarios.
Thanks,
-Max
On Sun, Oct 9, 2022 at 6:10 AM Gyula Fóra <gyula.f...@gmail.com> wrote:
Hi!
I think we have to make sure that the Rescale API will work also without
the adaptive scheduler (for instance when we are running Flink with the
Kubernetes Native Integration or in other cases where the adaptive
scheduler is not supported).
What do you think?
Cheers
Gyula
On Fri, Oct 7, 2022 at 3:50 PM Maximilian Michels <m...@apache.org> wrote:
We've been looking into ways to support programmatic rescaling of job
vertices. This feature is typically required for any type of Flink
autoscaler which does not merely set the default parallelism but adjusts
the parallelisms on a JobVertex level.
We've had an initial discussion here:
https://issues.apache.org/jira/browse/FLINK-29501 where Chesnay suggested
to use the infamous "rescaling" API:
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobs-jobid-rescaling
This API is disabled as of
https://issues.apache.org/jira/browse/FLINK-12312
.
Since there is the Adaptive Scheduler in Flink now, it may be feasible to
re-enable the API (at least for streaming jobs) and allow overriding the
parallelism of job vertices in addition to the default parallelism.
Any thoughts?
-Max
