Hi Jingsong, Thanks for the explanation about the mechanism of the new Flink session cluster mode.
Because I mostly use job cluster mode, so did not have a good knowledge of the new Flink session cluster mode. Best, Vino Jingsong Li <jingsongl...@gmail.com> 于2019年11月21日周四 下午2:46写道： > Hi Piper and Vino: > > Current Flink version, the resources of Flink Session cluster > are unrestricted, which means if the requested resources exceed the > resources owned by the current session, it will apply to the RM of yarn for > new resources. > And if TaskManager is idle for too long, JM will release it to yarn. This > behavior is controlled by resourcemanager.taskmanager-timeout . You can set > a suitable value for it to enjoy the benefits of reuse process and dynamic > resources. > > From this point of view, I think session mode is a good choice. > Is this what you want? Piper. > > Best, > Jingsong Lee > > > > On Thu, Nov 21, 2019 at 2:25 PM vino yang <yanghua1...@gmail.com> wrote: > >> Hi Piper, >> >> The understanding of two deploy modes For Flink on Yarn is right. >> >> AFAIK, The single job (job cluster) mode is more popular than Session >> mode. >> >> Because job cluster mode, Flink let YARN manage resources as far as >> possible. And this mode can keep isolation from other jobs. >> >> IMO, we do not need to combine their advantages. Let YARN do the things >> that it is good at. What do you think? >> >> Best, >> Vino >> >> >> Piper Piper <piperfl...@gmail.com> 于2019年11月21日周四 上午11:55写道： >> >>> Hi Vino, >>> >>> I want to implement Resource Elasticity. In doing so, I have read that >>> Flink with YARN has two modes: Job and Session. >>> >>> In Job mode, Flink’s Resource Manager requests YARN for containers with >>> TMs, and then gives the containers back to YARN upon job completion. >>> >>> In Session mode, Flink already has the TMs that are persistent. >>> >>> I want to combine the advantages of Job and Session mode, i.e. Flink >>> will have persistent TMs/containers and request YARN for more >>> TMs/containers when needed (or release TMs/containers back to YARN). >>> >>> Thank you, >>> >>> Piper >>> >>> On Wed, Nov 20, 2019 at 9:39 PM vino yang <yanghua1...@gmail.com> wrote: >>> >>>> Hi Piper, >>>> >>>> Can you share more reason and details of your requirements. >>>> >>>> Best, >>>> Vino >>>> >>>> Piper Piper <piperfl...@gmail.com> 于2019年11月21日周四 上午5:48写道： >>>> >>>>> Hi, >>>>> >>>>> How can I make Flink's Resource Manager request YARN to spin up new >>>>> (or destroy/reclaim existing) TaskManagers in YARN containers? >>>>> >>>>> Preferably at runtime (i.e. dynamically). >>>>> >>>>> Thank you >>>>> >>>>> Piper >>>>> >>>> > > -- > Best, Jingsong Lee >
