Hi everyone, I'm trying to setup Spark 1.5.1 with mesos and the Cluster Dispatcher that I'm currently running on one of the slaves. We're migrating from a 1.3 standalone cluster and we're hoping to benefit from dynamic resource allocation with fine grained mesos for a better distribution of resources between regular jobs and streaming jobs.
My driver tries to launch, but fails because none of the system properties are present, yet in the UI for the dispatcher it lists the options. It looks related to this: https://issues.apache.org/jira/browse/SPARK-2921 *Spark Submit* bin/spark-submit --deploy-mode cluster --master mesos:// > mesoss1.qa.xxx.com:7077 --conf > "spark.executor.extraJavaOptions=-Dcom.sun.jersey.server.impl.cdi.lookupExtensionInBeanManager=true" > --class com.xxx.tools.cdi.Bootstrap > /home/deployment/spork-1.0.0-SNAPSHOT-fat.jar /home/deployment/xyz.json also tried spark.driver.extraJavaOptions without success *Dispatcher UI* Scheduler propertyValuespark.executor.extraJavaOptions -Dcom.sun.jersey.server.impl.cdi.lookupExtensionInBeanManager=truespark.jars file:/home/deployment/spork-1.0.0-SNAPSHOT-fat.jarspark.driver.supervise falsespark.app.namecom.xxx.tools.cdi.Bootstrapspark.submit.deployModecluster spark.mastermesos://mesoss1.qa.xxx.com:7077 *Logs in Sandbox* Starting task driver-20151104023425-0011 > Forked command at 6554 > sh -c '/opt/spark/bin/spark-submit --name com.xxx.tools.cdi.Bootstrap > --class com.xxx.tools.cdi.Bootstrap --master > mesos://zk://zk-seed-1:2181,zk-seed-2:2181,zk-seed-3:2181/mesos > --driver-cores 1.0 --driver-memory 1024M spork-1.0.0-SNAPSHOT-fat.jar > /home/deployment/xyz.json' I hope someone has a clue. Thanks!! - Jo
