I have a spark app in development which has relatively strict cpu/mem ratios that are required. As such, I cannot arbitrarily add CPUs to a limited memory size.
The general spark cluster behaves as expected, where tasks are launched with a specified memory/cpu ratio, but the mesos scheduler seems to ignore this. Specifically, I cannot find where in the code the limit of number of tasks per executor of "spark.executor.cores" / "spark.task.cpus" is enforced in the MesosBackendScheduler. The Spark App in question has some JVM heap heavy activities inside a RDD.mapPartitionsWithIndex, so having more tasks per limited JVM memory resource is bad. The workaround planned handling of this is to limit the number of tasks per JVM, which does not seem possible in mesos mode, where it seems to just keep stacking on CPUs as tasks come in without adjusting any memory constraints, or looking for limits of tasks per executor. How can I limit the tasks per executor (or per memory pool) in the Mesos backend scheduler? Thanks, Charles Allen
