Hi Tim, I've done more forensics on this bug, see my comment here:
https://issues.apache.org/jira/browse/SPARK-11327?focusedCommentId=15009843&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15009843 - Jo Voordeckers On Tue, Nov 17, 2015 at 12:01 PM, Timothy Chen <tnac...@gmail.com> wrote: > Hi Jo, > > Thanks for the links, I would expected the properties to be in > scheduler properties but I need to double check. > > I'll be looking into these problems this week. > > Tim > > On Tue, Nov 17, 2015 at 10:28 AM, Jo Voordeckers > <jo.voordeck...@gmail.com> wrote: > > On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș < > iulian.dra...@typesafe.com> > > wrote: > >> > >> I think it actually tries to send all properties as part of > >> `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed: > >> > >> > >> > https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377 > >> > > > > Aha that's interesting, I overlooked that line, I'll debug some more > today > > because I know for sure that those options don't make it onto the > > commandline when I was running it in my debugger. > > > >> > >> Can you please open a Jira ticket and describe also the symptoms? This > >> might be related, or the same issue: SPARK-11280 and also SPARK-11327 > > > > > > SPARK-11327 is exactly my problem, but I don't run docker. > > > > - Jo > > > >> On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers < > jo.voordeck...@gmail.com> > >> wrote: > >>> > >>> > >>> Hi all, > >>> > >>> I'm running the mesos cluster dispatcher, however when I submit jobs > with > >>> things like jvm args, classpath order and UI port aren't added to the > >>> commandline executed by the mesos scheduler. In fact it only cares > about the > >>> class, jar and num cores/mem. > >>> > >>> > >>> > https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424 > >>> > >>> I've made an attempt at adding a few of the args that I believe are > >>> useful to the MesosClusterScheduler class, which seems to solve my > problem. > >>> > >>> Please have a look: > >>> > >>> https://github.com/apache/spark/pull/9752 > >>> > >>> Thanks > >>> > >>> - Jo Voordeckers > >>> > >> > >> > >> > >> -- > >> > >> -- > >> Iulian Dragos > >> > >> ------ > >> Reactive Apps on the JVM > >> www.typesafe.com > >> > > >
