Hi Jingsong, Thank you for your reply!
>Is this what you want? Piper. Yes. This is exactly what I want. Is there any way for me to specify to Flink RM how much of resources to ask YARN's RM for, and if we want Flink's RM to ask for resources proactively before it runs out? Similarly, is there any way I can force the JM to release TM back to YARN before timeout? Or will I need to modify the source code of Flink for this? Thank you, Piper On Thu, Nov 21, 2019 at 2:17 AM vino yang <yanghua1...@gmail.com> wrote: > Hi Jingsong, > > Thanks for the explanation about the mechanism of the new Flink session > cluster mode. > > Because I mostly use job cluster mode, so did not have a good knowledge of > the new Flink session cluster mode. > > Best, > Vino > > Jingsong Li <jingsongl...@gmail.com> 于2019年11月21日周四 下午2:46写道： > >> Hi Piper and Vino: >> >> Current Flink version, the resources of Flink Session cluster >> are unrestricted, which means if the requested resources exceed the >> resources owned by the current session, it will apply to the RM of yarn for >> new resources. >> And if TaskManager is idle for too long, JM will release it to yarn. This >> behavior is controlled by resourcemanager.taskmanager-timeout . You can set >> a suitable value for it to enjoy the benefits of reuse process and dynamic >> resources. >> >> From this point of view, I think session mode is a good choice. >> Is this what you want? Piper. >> >> Best, >> Jingsong Lee >> >> >> >> On Thu, Nov 21, 2019 at 2:25 PM vino yang <yanghua1...@gmail.com> wrote: >> >>> Hi Piper, >>> >>> The understanding of two deploy modes For Flink on Yarn is right. >>> >>> AFAIK, The single job (job cluster) mode is more popular than Session >>> mode. >>> >>> Because job cluster mode, Flink let YARN manage resources as far as >>> possible. And this mode can keep isolation from other jobs. >>> >>> IMO, we do not need to combine their advantages. Let YARN do the things >>> that it is good at. What do you think? >>> >>> Best, >>> Vino >>> >>> >>> Piper Piper <piperfl...@gmail.com> 于2019年11月21日周四 上午11:55写道： >>> >>>> Hi Vino, >>>> >>>> I want to implement Resource Elasticity. In doing so, I have read that >>>> Flink with YARN has two modes: Job and Session. >>>> >>>> In Job mode, Flink’s Resource Manager requests YARN for containers with >>>> TMs, and then gives the containers back to YARN upon job completion. >>>> >>>> In Session mode, Flink already has the TMs that are persistent. >>>> >>>> I want to combine the advantages of Job and Session mode, i.e. Flink >>>> will have persistent TMs/containers and request YARN for more >>>> TMs/containers when needed (or release TMs/containers back to YARN). >>>> >>>> Thank you, >>>> >>>> Piper >>>> >>>> On Wed, Nov 20, 2019 at 9:39 PM vino yang <yanghua1...@gmail.com> >>>> wrote: >>>> >>>>> Hi Piper, >>>>> >>>>> Can you share more reason and details of your requirements. >>>>> >>>>> Best, >>>>> Vino >>>>> >>>>> Piper Piper <piperfl...@gmail.com> 于2019年11月21日周四 上午5:48写道： >>>>> >>>>>> Hi, >>>>>> >>>>>> How can I make Flink's Resource Manager request YARN to spin up new >>>>>> (or destroy/reclaim existing) TaskManagers in YARN containers? >>>>>> >>>>>> Preferably at runtime (i.e. dynamically). >>>>>> >>>>>> Thank you >>>>>> >>>>>> Piper >>>>>> >>>>> >> >> -- >> Best, Jingsong Lee >> >
