Hi Yang,
Session mode is working exactly as you described. No exceptions.
Thank you!
Piper
On Sun, Nov 24, 2019 at 11:24 PM Yang Wang wrote:
> Hi Piper,
>
> In session mode, Flink will always use the free slots in the existing
> TaskManagers first.
> When it can not full fill the slot reques
Hi Piper,
In session mode, Flink will always use the free slots in the existing
TaskManagers first.
When it can not full fill the slot request, new TaskManagers will be
started.
Did you find some exceptions?
Best,
Yang
Piper Piper 于2019年11月23日周六 上午8:52写道：
> Hello Yang,
>
> Thank you for the ex
Hello Yang,
Thank you for the explanation!
I want to control the amount of TaskManagers in order to have finer control
over allowing/rejecting certain jobs in the cluster.
In Session mode with multiple jobs, is there any way to control whether
Flink will fit a new job into empty slots in existin
Hi Piper,
Jingsong is right. Both per-job and session cluster, the
YarnResourceManager will allocate
taskmanager containers dynamically on demand.
For per-job cluster, it will allocate taskmanagers base on the job slot
demand. The excess
containers will return to yarn immediately. When the job fi
Thank you, I will check it out.
On Thu, Nov 21, 2019, 9:21 PM Jingsong Li wrote:
> Hi Piper,
>
> AFAIK, There are no these flexible operations. You can get some
> information from metrics, but you can not control them.
> Maybe you should modify some source code in flink-yarn.
>
> Best,
> Jingson
Hi Piper,
AFAIK, There are no these flexible operations. You can get some information
from metrics, but you can not control them.
Maybe you should modify some source code in flink-yarn.
Best,
Jingsong Lee
On Thu, Nov 21, 2019 at 8:17 PM Piper Piper wrote:
> Hi Jingsong,
>
> Thank you for your
Hi Jingsong,
Thank you for your reply!
>Is this what you want? Piper.
Yes. This is exactly what I want.
Is there any way for me to specify to Flink RM how much of resources to ask
YARN's RM for, and if we want Flink's RM to ask for resources proactively
before it runs out?
Similarly, is there a
Hi Jingsong,
Thanks for the explanation about the mechanism of the new Flink session
cluster mode.
Because I mostly use job cluster mode, so did not have a good knowledge of
the new Flink session cluster mode.
Best,
Vino
Jingsong Li 于2019年11月21日周四 下午2:46写道：
> Hi Piper and Vino:
>
> Current Fl
Hi Piper and Vino:
Current Flink version, the resources of Flink Session cluster
are unrestricted, which means if the requested resources exceed the
resources owned by the current session, it will apply to the RM of yarn for
new resources.
And if TaskManager is idle for too long, JM will release i
Hi Piper,
The understanding of two deploy modes For Flink on Yarn is right.
AFAIK, The single job (job cluster) mode is more popular than Session mode.
Because job cluster mode, Flink let YARN manage resources as far as
possible. And this mode can keep isolation from other jobs.
IMO, we do not
Hi Vino,
I want to implement Resource Elasticity. In doing so, I have read that
Flink with YARN has two modes: Job and Session.
In Job mode, Flink’s Resource Manager requests YARN for containers with
TMs, and then gives the containers back to YARN upon job completion.
In Session mode, Flink alre
Hi Piper,
Can you share more reason and details of your requirements.
Best,
Vino
Piper Piper 于2019年11月21日周四 上午5:48写道：
> Hi,
>
> How can I make Flink's Resource Manager request YARN to spin up new (or
> destroy/reclaim existing) TaskManagers in YARN containers?
>
> Preferably at runtime (i.e. d
Hi,
How can I make Flink's Resource Manager request YARN to spin up new (or
destroy/reclaim existing) TaskManagers in YARN containers?
Preferably at runtime (i.e. dynamically).
Thank you
Piper
13 matches
Mail list logo
