Hi Piper, AFAIK, There are no these flexible operations. You can get some information from metrics, but you can not control them. Maybe you should modify some source code in flink-yarn.
Best, Jingsong Lee On Thu, Nov 21, 2019 at 8:17 PM Piper Piper <piperfl...@gmail.com> wrote: > Hi Jingsong, > > Thank you for your reply! > > >Is this what you want? Piper. > > Yes. This is exactly what I want. > > Is there any way for me to specify to Flink RM how much of resources to > ask YARN's RM for, and if we want Flink's RM to ask for resources > proactively before it runs out? > Similarly, is there any way I can force the JM to release TM back to YARN > before timeout? > > Or will I need to modify the source code of Flink for this? > > Thank you, > > Piper > > On Thu, Nov 21, 2019 at 2:17 AM vino yang <yanghua1...@gmail.com> wrote: > >> Hi Jingsong, >> >> Thanks for the explanation about the mechanism of the new Flink session >> cluster mode. >> >> Because I mostly use job cluster mode, so did not have a good knowledge >> of the new Flink session cluster mode. >> >> Best, >> Vino >> >> Jingsong Li <jingsongl...@gmail.com> 于2019年11月21日周四 下午2:46写道： >> >>> Hi Piper and Vino: >>> >>> Current Flink version, the resources of Flink Session cluster >>> are unrestricted, which means if the requested resources exceed the >>> resources owned by the current session, it will apply to the RM of yarn for >>> new resources. >>> And if TaskManager is idle for too long, JM will release it to yarn. >>> This behavior is controlled by resourcemanager.taskmanager-timeout . You >>> can set a suitable value for it to enjoy the benefits of reuse process and >>> dynamic resources. >>> >>> From this point of view, I think session mode is a good choice. >>> Is this what you want? Piper. >>> >>> Best, >>> Jingsong Lee >>> >>> >>> >>> On Thu, Nov 21, 2019 at 2:25 PM vino yang <yanghua1...@gmail.com> wrote: >>> >>>> Hi Piper, >>>> >>>> The understanding of two deploy modes For Flink on Yarn is right. >>>> >>>> AFAIK, The single job (job cluster) mode is more popular than Session >>>> mode. >>>> >>>> Because job cluster mode, Flink let YARN manage resources as far as >>>> possible. And this mode can keep isolation from other jobs. >>>> >>>> IMO, we do not need to combine their advantages. Let YARN do the things >>>> that it is good at. What do you think? >>>> >>>> Best, >>>> Vino >>>> >>>> >>>> Piper Piper <piperfl...@gmail.com> 于2019年11月21日周四 上午11:55写道： >>>> >>>>> Hi Vino, >>>>> >>>>> I want to implement Resource Elasticity. In doing so, I have read that >>>>> Flink with YARN has two modes: Job and Session. >>>>> >>>>> In Job mode, Flink’s Resource Manager requests YARN for containers >>>>> with TMs, and then gives the containers back to YARN upon job completion. >>>>> >>>>> In Session mode, Flink already has the TMs that are persistent. >>>>> >>>>> I want to combine the advantages of Job and Session mode, i.e. Flink >>>>> will have persistent TMs/containers and request YARN for more >>>>> TMs/containers when needed (or release TMs/containers back to YARN). >>>>> >>>>> Thank you, >>>>> >>>>> Piper >>>>> >>>>> On Wed, Nov 20, 2019 at 9:39 PM vino yang <yanghua1...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Piper, >>>>>> >>>>>> Can you share more reason and details of your requirements. >>>>>> >>>>>> Best, >>>>>> Vino >>>>>> >>>>>> Piper Piper <piperfl...@gmail.com> 于2019年11月21日周四 上午5:48写道： >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> How can I make Flink's Resource Manager request YARN to spin up new >>>>>>> (or destroy/reclaim existing) TaskManagers in YARN containers? >>>>>>> >>>>>>> Preferably at runtime (i.e. dynamically). >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> Piper >>>>>>> >>>>>> >>> >>> -- >>> Best, Jingsong Lee >>> >> -- Best, Jingsong Lee
