Hi MaGuoWei, would you like to have done automatically by Flink or based on some user input? Adding commands to the ./bin/yarn-session.sh to change the cluster size is quite easy. However, reducing the cluster size while a job is running will fail the job.
Making this automatically is much harder. On Mon, Jul 27, 2015 at 12:07 PM, Maximilian Michels <m...@apache.org> wrote: > Hi MaGuoWei, > > If I understand correctly, you're are looking for a way to have a job > manager (master) contentiously running which requests YARN containers for > the task managers (workers) on the fly. This is currently not supported by > Flink although you can add or remove task managers while the cluster is > running. The job manager distributes task to the task manager which are > available at scheduling time. > > As Aljoscha mentioned, the most flexible way of sharing resources in a YARN > environment is to start a per-job cluster for each job. > > Cheers, > Max > > On Mon, Jul 27, 2015 at 11:19 AM, MaGuoWei <maguo...@outlook.com> wrote: > > > It is great! Is there any document? I am very interested in this. > > thanks > > > > > > > From: aljos...@apache.org > > > Date: Mon, 27 Jul 2015 05:14:00 +0000 > > > Subject: Re: add some new api to the scheduler in the job manager > > > To: dev@flink.apache.org > > > > > > Hi, > > > I think for more details on giving back resources of a running cluster > we > > > have to wait for Robert's opinion. In the mean time, you can also just > > run > > > a single job that will bring up some yarn containers and then release > > them > > > afterward using this: > > > > > > https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html#run-a-single-flink-job-on-hadoop-yarn > > > > > > Cheers, > > > Aljoscha > > > > > > On Mon, 27 Jul 2015 at 03:51 MaGuoWei <maguo...@outlook.com> wrote: > > > > > > > hi guysFlink must take over all the resources all the time. That > wastes > > > > resources sometime especially in the sharing cluster.For example > when > > > > using Flink on Yarn the resource can't be returned back to RM even if > > no > > > > graph is running.So I want to know: Is it possible to add some apis > in > > the > > > > scheduler to request the resource(slot) and release the > resource(slot)? > > > > These apis can be customized in the different circumstance. > > > > thanks a lot! > > > > > > > > > > > > > > > > >
