Re: Specifying number of executors in Mesos

Gerard, Are you familiar with spark.deploy.spreadOut in Standalone mode? It sounds like you want the same thing in Mesos mode. On Thu, Dec 11, 2014 at 6:48 AM, Tathagata Das wrote: > Not that I am aware of. Spark will try to spread th

Re: Specifying number of executors in Mesos

Not that I am aware of. Spark will try to spread the tasks evenly across executors, its not aware of the workers at all. So if the executors to worker allocation is uneven, I am not sure what can be done. Maybe others can get smoe ideas. On Tue, Dec 9, 2014 at 6:20 AM, Gerard Maas wrote: > Hi, >

Specifying number of executors in Mesos

Hi, We've a number of Spark Streaming /Kafka jobs that would benefit of an even spread of consumers over physical hosts in order to maximize network usage. As far as I can see, the Spark Mesos scheduler accepts resource offers until all required Mem + CPU allocation has been satisfied. This basic