thanks all you guys.Now I know I can achieve this goal by creating a cluster per topology and estimating the cluster size by analyzing the JobGraph(or by some user input).But I think it can be more beautiful if Flink can abstract some common resource api such as get/release/deploy. :-) thanks.
> From: rmetz...@apache.org > Date: Mon, 27 Jul 2015 14:01:11 +0200 > Subject: Re: add some new api to the scheduler in the job manager > To: dev@flink.apache.org > > Hi MaGuoWei, > > would you like to have done automatically by Flink or based on some user > input? > Adding commands to the ./bin/yarn-session.sh to change the cluster size is > quite easy. However, reducing the cluster size while a job is running will > fail the job. > > Making this automatically is much harder. > > On Mon, Jul 27, 2015 at 12:07 PM, Maximilian Michels <m...@apache.org> wrote: > > > Hi MaGuoWei, > > > > If I understand correctly, you're are looking for a way to have a job > > manager (master) contentiously running which requests YARN containers for > > the task managers (workers) on the fly. This is currently not supported by > > Flink although you can add or remove task managers while the cluster is > > running. The job manager distributes task to the task manager which are > > available at scheduling time. > > > > As Aljoscha mentioned, the most flexible way of sharing resources in a YARN > > environment is to start a per-job cluster for each job. > > > > Cheers, > > Max > > > > On Mon, Jul 27, 2015 at 11:19 AM, MaGuoWei <maguo...@outlook.com> wrote: > > > > > It is great! Is there any document? I am very interested in this. > > > thanks > > > > > > > > > > From: aljos...@apache.org > > > > Date: Mon, 27 Jul 2015 05:14:00 +0000 > > > > Subject: Re: add some new api to the scheduler in the job manager > > > > To: dev@flink.apache.org > > > > > > > > Hi, > > > > I think for more details on giving back resources of a running cluster > > we > > > > have to wait for Robert's opinion. In the mean time, you can also just > > > run > > > > a single job that will bring up some yarn containers and then release > > > them > > > > afterward using this: > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html#run-a-single-flink-job-on-hadoop-yarn > > > > > > > > Cheers, > > > > Aljoscha > > > > > > > > On Mon, 27 Jul 2015 at 03:51 MaGuoWei <maguo...@outlook.com> wrote: > > > > > > > > > hi guysFlink must take over all the resources all the time. That > > wastes > > > > > resources sometime especially in the sharing cluster.For example > > when > > > > > using Flink on Yarn the resource can't be returned back to RM even if > > > no > > > > > graph is running.So I want to know: Is it possible to add some apis > > in > > > the > > > > > scheduler to request the resource(slot) and release the > > resource(slot)? > > > > > These apis can be customized in the different circumstance. > > > > > thanks a lot! > > > > > > > > > > > > > > > > > > > > > > >
