Hi, Nagaraj: Thanks for the response, but this does not solve my problem. I think executor memory should be proportional to number of cores, or number of core in each executor should be the same. On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar < [email protected]> wrote:
> Hi Renjie, > > I have not setup Spark Streaming on Mesos but there is something called > reservations in Mesos. It supports both Static and Dynamic reservations. > Both types of reservations must have role defined. You may want to explore > these options. Excerpts from the Apache Mesos documentation. > > Cheers > Nagaraj C > Reservation > > Mesos provides mechanisms to reserve resources in specific slaves. The > concept was first introduced with static reservation in 0.14.0 which > enabled operators to specify the reserved resources on slave startup. This > was extended with dynamic reservation in 0.23.0 which enabled operators > and authorized frameworks to dynamically reserve resources in the cluster. > > No breaking changes were introduced with dynamic reservation, which means > the existing static reservation mechanism continues to be fully supported. > > In both types of reservations, resources are reserved for a role. > Static Reservation (since 0.14.0) > > An operator can configure a slave with resources reserved for a role. The > reserved resources are specified via the --resources flag. For example, > suppose we have 12 CPUs and 6144 MB of RAM available on a slave and that we > want to reserve 8 CPUs and 4096 MB of RAM for the ads role. We start the > slave like so: > > $ mesos-slave \ > --master=<ip>:<port> \ > --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096" > > We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave. > > > From: Renjie Liu <[email protected]> > Date: Friday, November 27, 2015 at 9:57 PM > To: "[email protected]" <[email protected]> > Subject: Spark Streaming on mesos > > Hi, all: > I'm trying to run spark streaming on mesos and it seems that none of the > scheduler is suitable for that. Fine grain scheduler will start an executor > for each task so it will significantly increase the latency. While coarse > grained mode can only set the max core numbers and executor memory but > there's no way to set the number of cores for each executor. Has anyone > deployed spark streaming on mesos? And what's your settings? > -- > Liu, Renjie > Software Engineer, MVAD > -- Liu, Renjie Software Engineer, MVAD
