Thanks a lot for all your input. I have update the FLIP-160 with your
suggestions:
1) Add job configuration as a follow up
2) Pull out IO operations out of the ExecutionGraph if the failover becomes
too slow
3) Introduce a configuration parameter for the timeout in the "Waiting for
resources" stat
Yes, since we're only operating within the scheduler, which exists
separately for each job, we don't have to worry about collisions with
other jobs.
On 1/27/2021 11:08 AM, Yangze Guo wrote:
Thanks for preparing the FLIP and driving the discussion, Till. All of
my questions have already been an
Thanks for preparing the FLIP and driving the discussion, Till. All of
my questions have already been answered in the previous discussion.
I just have one minor reminder regarding using ExecutionVertexID as
the identificator. The JobVertexID is generated according to the
topology instead of genera
Thanks for the explanations, Till.
Keeping the initial design as simple as possible sounds good to me.
There's no further concern from my side.
Thank you~
Xintong Song
On Tue, Jan 26, 2021 at 9:56 PM Zhu Zhu wrote:
> Thanks Till for the explanation and the follow up actions.
> That sounds go
Thanks Till for the explanation and the follow up actions.
That sounds good to me.
Thanks,
Zhu
Till Rohrmann 于2021年1月26日周二 下午7:28写道：
> Thanks a lot for all your feedback. Let me try to answer it.
>
> # ScaleUpController vs. RescaleController
>
> At the moment, the idea of the declarative schedu
Thanks a lot for all your feedback. Let me try to answer it.
# ScaleUpController vs. RescaleController
At the moment, the idea of the declarative scheduler is to run a job with a
parallelism which is as close to the target value as possible but to never
exceed it. Since the target value is curren
Thanks for the proposal! @Till Rohrmann
The design looks generally good to me. I have 2 concerns though:
# performance on failover
I can see is that an ExecutionGraph will be built and initialized on each
task failure. The process is possibly to be slow:
1. the topology building can be very slow
Thanks for preparing the FLIP and starting the discussion, Till.
I have a few questions trying to understand the design.
## What is the relationship between the current and new schedulers?
IIUC, the new declarative scheduler will coexist with the current
scheduler, as an alternative that the user
Till, thanks a lot for the proposal.
Even if the initial phase is only to support scale-up, maybe the
"ScaleUpController" interface should be called "RescaleController" so that
in the future scale-down can be added.
On Fri, Jan 22, 2021 at 7:03 AM Till Rohrmann wrote:
> Hi everyone,
>
> I would
Hi everyone,
I would like to start a discussion about adding a new type of scheduler to
Flink. The declarative scheduler will first declare the required resources
and wait for them before deciding on the actual parallelism of a job.
Thereby it can better handle situations where resources cannot be
10 matches
Mail list logo
