Hi Jo, I agree that there's something fishy with the cluster dispatcher, I've seen some issues like that.
I think it actually tries to send all properties as part of `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed: https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377 Can you please open a Jira ticket and describe also the symptoms? This might be related, or the same issue: SPARK-11280 <https://issues.apache.org/jira/browse/SPARK-11280> and also SPARK-11327 <https://issues.apache.org/jira/browse/SPARK-11327> thanks, iulian On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <jo.voordeck...@gmail.com> wrote: > > Hi all, > > I'm running the mesos cluster dispatcher, however when I submit jobs with > things like jvm args, classpath order and UI port aren't added to the > commandline executed by the mesos scheduler. In fact it only cares about > the class, jar and num cores/mem. > > > https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424 > > I've made an attempt at adding a few of the args that I believe are useful > to the MesosClusterScheduler class, which seems to solve my problem. > > Please have a look: > > https://github.com/apache/spark/pull/9752 > > Thanks > > - Jo Voordeckers > > -- -- Iulian Dragos ------ Reactive Apps on the JVM www.typesafe.com
