I think what MaGuoWei is looking for is "intra-job" YARN elasticity.
Adding these hooks is going to be important also to trigger streaming jobs to scale in/out during runtime. On Tue, Jul 28, 2015 at 3:27 AM, MaGuoWei <maguo...@outlook.com> wrote: > thanks all you guys.Now I know I can achieve this goal by creating a > cluster per topology and estimating the cluster size by analyzing the > JobGraph(or by some user input).But I think it can be more beautiful if > Flink can abstract some common resource api such as get/release/deploy. :-) > thanks. > > > > > From: rmetz...@apache.org > > Date: Mon, 27 Jul 2015 14:01:11 +0200 > > Subject: Re: add some new api to the scheduler in the job manager > > To: dev@flink.apache.org > > > > Hi MaGuoWei, > > > > would you like to have done automatically by Flink or based on some user > > input? > > Adding commands to the ./bin/yarn-session.sh to change the cluster size > is > > quite easy. However, reducing the cluster size while a job is running > will > > fail the job. > > > > Making this automatically is much harder. > > > > On Mon, Jul 27, 2015 at 12:07 PM, Maximilian Michels <m...@apache.org> > wrote: > > > > > Hi MaGuoWei, > > > > > > If I understand correctly, you're are looking for a way to have a job > > > manager (master) contentiously running which requests YARN containers > for > > > the task managers (workers) on the fly. This is currently not > supported by > > > Flink although you can add or remove task managers while the cluster is > > > running. The job manager distributes task to the task manager which are > > > available at scheduling time. > > > > > > As Aljoscha mentioned, the most flexible way of sharing resources in a > YARN > > > environment is to start a per-job cluster for each job. > > > > > > Cheers, > > > Max > > > > > > On Mon, Jul 27, 2015 at 11:19 AM, MaGuoWei <maguo...@outlook.com> > wrote: > > > > > > > It is great! Is there any document? I am very interested in this. > > > > thanks > > > > > > > > > > > > > From: aljos...@apache.org > > > > > Date: Mon, 27 Jul 2015 05:14:00 +0000 > > > > > Subject: Re: add some new api to the scheduler in the job manager > > > > > To: dev@flink.apache.org > > > > > > > > > > Hi, > > > > > I think for more details on giving back resources of a running > cluster > > > we > > > > > have to wait for Robert's opinion. In the mean time, you can also > just > > > > run > > > > > a single job that will bring up some yarn containers and then > release > > > > them > > > > > afterward using this: > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html#run-a-single-flink-job-on-hadoop-yarn > > > > > > > > > > Cheers, > > > > > Aljoscha > > > > > > > > > > On Mon, 27 Jul 2015 at 03:51 MaGuoWei <maguo...@outlook.com> > wrote: > > > > > > > > > > > hi guysFlink must take over all the resources all the time. That > > > wastes > > > > > > resources sometime especially in the sharing cluster.For example > > > when > > > > > > using Flink on Yarn the resource can't be returned back to RM > even if > > > > no > > > > > > graph is running.So I want to know: Is it possible to add some > apis > > > in > > > > the > > > > > > scheduler to request the resource(slot) and release the > > > resource(slot)? > > > > > > These apis can be customized in the different circumstance. > > > > > > thanks a lot! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
