Thank you, I will check it out. On Thu, Nov 21, 2019, 9:21 PM Jingsong Li <jingsongl...@gmail.com> wrote:
> Hi Piper, > > AFAIK, There are no these flexible operations. You can get some > information from metrics, but you can not control them. > Maybe you should modify some source code in flink-yarn. > > Best, > Jingsong Lee > > > On Thu, Nov 21, 2019 at 8:17 PM Piper Piper <piperfl...@gmail.com> wrote: > >> Hi Jingsong, >> >> Thank you for your reply! >> >> >Is this what you want? Piper. >> >> Yes. This is exactly what I want. >> >> Is there any way for me to specify to Flink RM how much of resources to >> ask YARN's RM for, and if we want Flink's RM to ask for resources >> proactively before it runs out? >> Similarly, is there any way I can force the JM to release TM back to YARN >> before timeout? >> >> Or will I need to modify the source code of Flink for this? >> >> Thank you, >> >> Piper >> >> On Thu, Nov 21, 2019 at 2:17 AM vino yang <yanghua1...@gmail.com> wrote: >> >>> Hi Jingsong, >>> >>> Thanks for the explanation about the mechanism of the new Flink session >>> cluster mode. >>> >>> Because I mostly use job cluster mode, so did not have a good knowledge >>> of the new Flink session cluster mode. >>> >>> Best, >>> Vino >>> >>> Jingsong Li <jingsongl...@gmail.com> 于2019年11月21日周四 下午2:46写道： >>> >>>> Hi Piper and Vino: >>>> >>>> Current Flink version, the resources of Flink Session cluster >>>> are unrestricted, which means if the requested resources exceed the >>>> resources owned by the current session, it will apply to the RM of yarn for >>>> new resources. >>>> And if TaskManager is idle for too long, JM will release it to yarn. >>>> This behavior is controlled by resourcemanager.taskmanager-timeout . You >>>> can set a suitable value for it to enjoy the benefits of reuse process and >>>> dynamic resources. >>>> >>>> From this point of view, I think session mode is a good choice. >>>> Is this what you want? Piper. >>>> >>>> Best, >>>> Jingsong Lee >>>> >>>> >>>> >>>> On Thu, Nov 21, 2019 at 2:25 PM vino yang <yanghua1...@gmail.com> >>>> wrote: >>>> >>>>> Hi Piper, >>>>> >>>>> The understanding of two deploy modes For Flink on Yarn is right. >>>>> >>>>> AFAIK, The single job (job cluster) mode is more popular than Session >>>>> mode. >>>>> >>>>> Because job cluster mode, Flink let YARN manage resources as far as >>>>> possible. And this mode can keep isolation from other jobs. >>>>> >>>>> IMO, we do not need to combine their advantages. Let YARN do the >>>>> things that it is good at. What do you think? >>>>> >>>>> Best, >>>>> Vino >>>>> >>>>> >>>>> Piper Piper <piperfl...@gmail.com> 于2019年11月21日周四 上午11:55写道： >>>>> >>>>>> Hi Vino, >>>>>> >>>>>> I want to implement Resource Elasticity. In doing so, I have read >>>>>> that Flink with YARN has two modes: Job and Session. >>>>>> >>>>>> In Job mode, Flink’s Resource Manager requests YARN for containers >>>>>> with TMs, and then gives the containers back to YARN upon job completion. >>>>>> >>>>>> In Session mode, Flink already has the TMs that are persistent. >>>>>> >>>>>> I want to combine the advantages of Job and Session mode, i.e. Flink >>>>>> will have persistent TMs/containers and request YARN for more >>>>>> TMs/containers when needed (or release TMs/containers back to YARN). >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Piper >>>>>> >>>>>> On Wed, Nov 20, 2019 at 9:39 PM vino yang <yanghua1...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Piper, >>>>>>> >>>>>>> Can you share more reason and details of your requirements. >>>>>>> >>>>>>> Best, >>>>>>> Vino >>>>>>> >>>>>>> Piper Piper <piperfl...@gmail.com> 于2019年11月21日周四 上午5:48写道： >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> How can I make Flink's Resource Manager request YARN to spin up new >>>>>>>> (or destroy/reclaim existing) TaskManagers in YARN containers? >>>>>>>> >>>>>>>> Preferably at runtime (i.e. dynamically). >>>>>>>> >>>>>>>> Thank you >>>>>>>> >>>>>>>> Piper >>>>>>>> >>>>>>> >>>> >>>> -- >>>> Best, Jingsong Lee >>>> >>> > > -- > Best, Jingsong Lee >
