Consider the spark.max.cores configuration option -- it should do what you require.
On Tue, Aug 11, 2015 at 8:26 AM, Haripriya Ayyalasomayajula < aharipriy...@gmail.com> wrote: > Hello all, > > As a quick follow up for this, I have been using Spark on Yarn till now > and am currently exploring Mesos and Marathon. Using yarn, we could tell > the spark job about the number of executors and number of cores as well, is > there a way to do it on mesos? I'm using Spark 1.4.1 on Mesos 0.23.0 and > Marathon 0.9. When we launch a marathon app, is there a way we can tell it > the max number of cores per executor ( which comes down to maximum number > of tasks per executor (for each instance of app)) - please correct me if I > am wrong. > > - I have been browsing over various documentation details but did not come > across a direct solution. > > - Is there a JIRA issue in progress already/a bug fix for the same? > > I greatly appreciate any help and would love to follow up/investigate > further if you can suggest if there is any JIRA issue already or any > pointers.. > > > On Wed, May 20, 2015 at 8:27 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> To put this on the devs' radar, I suggest creating a JIRA for it (and >> checking first if one already exists). >> >> issues.apache.org/jira/ >> >> Nick >> >> On Tue, May 19, 2015 at 1:34 PM Matei Zaharia <matei.zaha...@gmail.com> >> wrote: >> >>> Yeah, this definitely seems useful there. There might also be some ways >>> to cap the application in Mesos, but I'm not sure. >>> >>> Matei >>> >>> On May 19, 2015, at 1:11 PM, Thomas Dudziak <tom...@gmail.com> wrote: >>> >>> I'm using fine-grained for a multi-tenant environment which is why I >>> would welcome the limit of tasks per job :) >>> >>> cheers, >>> Tom >>> >>> On Tue, May 19, 2015 at 10:05 AM, Matei Zaharia <matei.zaha...@gmail.com >>> > wrote: >>> >>>> Hey Tom, >>>> >>>> Are you using the fine-grained or coarse-grained scheduler? For the >>>> coarse-grained scheduler, there is a spark.cores.max config setting that >>>> will limit the total # of cores it grabs. This was there in earlier >>>> versions too. >>>> >>>> Matei >>>> >>>> > On May 19, 2015, at 12:39 PM, Thomas Dudziak <tom...@gmail.com> >>>> wrote: >>>> > >>>> > I read the other day that there will be a fair number of improvements >>>> in 1.4 for Mesos. Could I ask for one more (if it isn't already in there): >>>> a configurable limit for the number of tasks for jobs run on Mesos ? This >>>> would be a very simple yet effective way to prevent a job dominating the >>>> cluster. >>>> > >>>> > cheers, >>>> > Tom >>>> > >>>> >>>> >>> >>> > > > -- > Regards, > Haripriya Ayyalasomayajula > >
