To properly abstract the resource manager behind an interface, it would be good to see what the common overlap of the YARN integration, and the pending mesos integration is.
Can anyone working on this chime in? On Tue, Jul 28, 2015 at 1:08 PM, Stephan Ewen <se...@apache.org> wrote: > I think what MaGuoWei is looking for is "intra-job" YARN elasticity. > > Adding these hooks is going to be important also to trigger streaming jobs > to scale in/out during runtime. > > On Tue, Jul 28, 2015 at 3:27 AM, MaGuoWei <maguo...@outlook.com> wrote: > >> thanks all you guys.Now I know I can achieve this goal by creating a >> cluster per topology and estimating the cluster size by analyzing the >> JobGraph(or by some user input).But I think it can be more beautiful if >> Flink can abstract some common resource api such as get/release/deploy. :-) >> thanks. >> >> >> >> > From: rmetz...@apache.org >> > Date: Mon, 27 Jul 2015 14:01:11 +0200 >> > Subject: Re: add some new api to the scheduler in the job manager >> > To: dev@flink.apache.org >> > >> > Hi MaGuoWei, >> > >> > would you like to have done automatically by Flink or based on some user >> > input? >> > Adding commands to the ./bin/yarn-session.sh to change the cluster size >> is >> > quite easy. However, reducing the cluster size while a job is running >> will >> > fail the job. >> > >> > Making this automatically is much harder. >> > >> > On Mon, Jul 27, 2015 at 12:07 PM, Maximilian Michels <m...@apache.org> >> wrote: >> > >> > > Hi MaGuoWei, >> > > >> > > If I understand correctly, you're are looking for a way to have a job >> > > manager (master) contentiously running which requests YARN containers >> for >> > > the task managers (workers) on the fly. This is currently not >> supported by >> > > Flink although you can add or remove task managers while the cluster >> is >> > > running. The job manager distributes task to the task manager which >> are >> > > available at scheduling time. >> > > >> > > As Aljoscha mentioned, the most flexible way of sharing resources in >> a YARN >> > > environment is to start a per-job cluster for each job. >> > > >> > > Cheers, >> > > Max >> > > >> > > On Mon, Jul 27, 2015 at 11:19 AM, MaGuoWei <maguo...@outlook.com> >> wrote: >> > > >> > > > It is great! Is there any document? I am very interested in this. >> > > > thanks >> > > > >> > > > >> > > > > From: aljos...@apache.org >> > > > > Date: Mon, 27 Jul 2015 05:14:00 +0000 >> > > > > Subject: Re: add some new api to the scheduler in the job manager >> > > > > To: dev@flink.apache.org >> > > > > >> > > > > Hi, >> > > > > I think for more details on giving back resources of a running >> cluster >> > > we >> > > > > have to wait for Robert's opinion. In the mean time, you can also >> just >> > > > run >> > > > > a single job that will bring up some yarn containers and then >> release >> > > > them >> > > > > afterward using this: >> > > > > >> > > > >> > > >> https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html#run-a-single-flink-job-on-hadoop-yarn >> > > > > >> > > > > Cheers, >> > > > > Aljoscha >> > > > > >> > > > > On Mon, 27 Jul 2015 at 03:51 MaGuoWei <maguo...@outlook.com> >> wrote: >> > > > > >> > > > > > hi guysFlink must take over all the resources all the time. That >> > > wastes >> > > > > > resources sometime especially in the sharing cluster.For >> example >> > > when >> > > > > > using Flink on Yarn the resource can't be returned back to RM >> even if >> > > > no >> > > > > > graph is running.So I want to know: Is it possible to add some >> apis >> > > in >> > > > the >> > > > > > scheduler to request the resource(slot) and release the >> > > resource(slot)? >> > > > > > These apis can be customized in the different circumstance. >> > > > > > thanks a lot! >> > > > > > >> > > > > > >> > > > > > >> > > > >> > > > >> > > >> >> > >
