Hi Renjie,
I have not setup Spark Streaming on Mesos but there is something called
reservations in Mesos. It supports both Static and Dynamic reservations. Both
types of reservations must have role defined. You may want to explore these
options. Excerpts from the Apache Mesos documentation.
Cheers
Nagaraj C
Reservation
Mesos provides mechanisms to reserve resources in specific slaves. The concept
was first introduced with static reservation in 0.14.0 which enabled operators
to specify the reserved resources on slave startup. This was extended with
dynamic reservation in 0.23.0 which enabled operators and authorized frameworks
to dynamically reserve resources in the cluster.
No breaking changes were introduced with dynamic reservation, which means the
existing static reservation mechanism continues to be fully supported.
In both types of reservations, resources are reserved for a role.
Static Reservation (since 0.14.0)
An operator can configure a slave with resources reserved for a role. The
reserved resources are specified via the --resources flag. For example, suppose
we have 12 CPUs and 6144 MB of RAM available on a slave and that we want to
reserve 8 CPUs and 4096 MB of RAM for the ads role. We start the slave like so:
$ mesos-slave \
--master=<ip>:<port> \
--resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096"
We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave.
From: Renjie Liu <[email protected]<mailto:[email protected]>>
Date: Friday, November 27, 2015 at 9:57 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Spark Streaming on mesos
Hi, all:
I'm trying to run spark streaming on mesos and it seems that none of the
scheduler is suitable for that. Fine grain scheduler will start an executor for
each task so it will significantly increase the latency. While coarse grained
mode can only set the max core numbers and executor memory but there's no way
to set the number of cores for each executor. Has anyone deployed spark
streaming on mesos? And what's your settings?
--
Liu, Renjie
Software Engineer, MVAD
