Hi Tim ,
I think I know the problem but i do not have a solution , *The Mesos Slave supposed to download the Jars from the URI specified and placed in $MESOS_SANDBOX location but it is not downloading not sure why* .. see below logs My command looks like below docker run -it --rm -m 2g -e SPARK_MASTER="mesos://10.0.2.15:7077" -e SPARK_IMAGE="spark_driver:latest" spark_driver:latest ./bin/spark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar [root@Mindstorm spark-1.6.0]# docker logs d22d8e897b79 *Warning: Local jar /mnt/mesos/sandbox/spark-examples-1.6.0-hadoop2.6.0.jar does not exist, skipping.* java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:278) at org.apache.spark.util.Utils$.classForName(Utils.scala:174) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) When i do docker inspect i see below command gets issued "Cmd": [ "-c", "./bin/spark-submit --name org.apache.spark.examples.SparkPi --master mesos://10.0.2.15:5050 --driver-cores 1.0 --driver-memory 1024M --class org.apache.spark.examples.SparkPi $*MESOS_SANDBOX*/spark-examples-1.6.0-hadoop2.6.0.jar " On Thu, Mar 3, 2016 at 12:09 AM, Tim Chen <t...@mesosphere.io> wrote: > You shouldn't need to specify --jars at all since you only have one jar. > > The error is pretty odd as it suggests it's trying to load > /opt/spark/Example but that doesn't really seem to be anywhere in your > image or command. > > Can you paste your stdout from the driver task launched by the cluster > dispatcher, that shows you the spark-submit command it eventually ran? > > > Tim > > > > On Wed, Mar 2, 2016 at 5:42 PM, Ashish Soni <asoni.le...@gmail.com> wrote: > >> See below and Attached the Dockerfile to build the spark image ( >> between i just upgraded to 1.6 ) >> >> I am running below setup - >> >> Mesos Master - Docker Container >> Mesos Slave 1 - Docker Container >> Mesos Slave 2 - Docker Container >> Marathon - Docker Container >> Spark MESOS Dispatcher - Docker Container >> >> when i submit the Spark PI Example Job using below command >> >> *docker run -it --rm -m 2g -e SPARK_MASTER="mesos://10.0.2.15:7077 >> <http://10.0.2.15:7077>" -e SPARK_IMAGE="spark_driver:**latest" >> spark_driver:latest ./bin/spark-submit --deploy-mode cluster --name "PI >> Example" --class org.apache.spark.examples.**SparkPi >> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >> <http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar> --jars >> /opt/spark/lib/spark-examples-**1.6.0-hadoop2.6.0.jar --verbose* >> >> Below is the ERROR >> Error: Cannot load main class from JAR file:/opt/spark/Example >> Run with --help for usage help or --verbose for debug output >> >> >> When i docker Inspect for the stopped / dead container i see below output >> what is interesting to see is some one or executor replaced by original >> command with below in highlighted and i do not see Executor is downloading >> the JAR -- IS this a BUG i am hitting or not sure if that is supposed to >> work this way and i am missing some configuration >> >> "Env": [ >> "SPARK_IMAGE=spark_driver:latest", >> "SPARK_SCALA_VERSION=2.10", >> "SPARK_VERSION=1.6.0", >> "SPARK_EXECUTOR_URI= >> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-hadoop2.6.tgz";, >> "MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos-0.25.0.so", >> "SPARK_MASTER=mesos://10.0.2.15:7077", >> >> "SPARK_EXECUTOR_OPTS=-Dspark.executorEnv.MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/ >> libmesos-0.25.0.so -Dspark.jars= >> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >> -Dspark.mesos.mesosExecutor.cores=0.1 -Dspark.driver.supervise=false - >> Dspark.app.name=PI Example -Dspark.mesos.uris= >> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >> -Dspark.mesos.executor.docker.image=spark_driver:latest >> -Dspark.submit.deployMode=cluster -Dspark.master=mesos://10.0.2.15:7077 >> -Dspark.driver.extraClassPath=/opt/spark/custom/lib/* >> -Dspark.executor.extraClassPath=/opt/spark/custom/lib/* >> -Dspark.executor.uri= >> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-hadoop2.6.tgz >> -Dspark.mesos.executor.home=/opt/spark", >> "MESOS_SANDBOX=/mnt/mesos/sandbox", >> >> "MESOS_CONTAINER_NAME=mesos-e47f8d4c-5ee1-4d01-ad07-0d9a03ced62d-S1.43c08f82-e508-4d57-8c0b-fa05bee77fd6", >> >> "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", >> "HADOOP_VERSION=2.6", >> "SPARK_HOME=/opt/spark" >> ], >> "Cmd": [ >> "-c", >> * "./bin/spark-submit --name PI Example --master >> mesos://10.0.2.15:5050 <http://10.0.2.15:5050> --driver-cores 1.0 >> --driver-memory 1024M --class org.apache.spark.examples.SparkPi >> $MESOS_SANDBOX/spark-examples-1.6.0-hadoop2.6.0.jar --jars >> /opt/spark/lib/spark-examples-1.6.0-hadoop2.6.0.jar --verbose"* >> ], >> "Image": "spark_driver:latest", >> >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Mar 2, 2016 at 5:49 PM, Charles Allen < >> charles.al...@metamarkets.com> wrote: >> >>> @Tim yes, this is asking about 1.5 though >>> >>> On Wed, Mar 2, 2016 at 2:35 PM Tim Chen <t...@mesosphere.io> wrote: >>> >>>> Hi Charles, >>>> >>>> I thought that's fixed with your patch in latest master now right? >>>> >>>> Ashish, yes please give me your docker image name (if it's in the >>>> public registry) and what you've tried and I can see what's wrong. I think >>>> it's most likely just the configuration of where the Spark home folder is >>>> in the image. >>>> >>>> Tim >>>> >>>> On Wed, Mar 2, 2016 at 2:28 PM, Charles Allen < >>>> charles.al...@metamarkets.com> wrote: >>>> >>>>> Re: Spark on Mesos.... Warning regarding disk space: >>>>> https://issues.apache.org/jira/browse/SPARK-12330 >>>>> >>>>> That's a spark flaw I encountered on a very regular basis on mesos. >>>>> That and a few other annoyances are fixed in >>>>> https://github.com/metamx/spark/tree/v1.5.2-mmx >>>>> >>>>> Here's another mild annoyance I've encountered: >>>>> https://issues.apache.org/jira/browse/SPARK-11714 >>>>> >>>>> On Wed, Mar 2, 2016 at 1:31 PM Ashish Soni <asoni.le...@gmail.com> >>>>> wrote: >>>>> >>>>>> I have no luck and i would to ask the question to spark committers >>>>>> will this be ever designed to run on mesos ? >>>>>> >>>>>> spark app as a docker container not working at all on mesos ,if any >>>>>> one would like the code i can send it over to have a look. >>>>>> >>>>>> Ashish >>>>>> >>>>>> On Wed, Mar 2, 2016 at 12:23 PM, Sathish Kumaran Vairavelu < >>>>>> vsathishkuma...@gmail.com> wrote: >>>>>> >>>>>>> Try passing jar using --jars option >>>>>>> >>>>>>> On Wed, Mar 2, 2016 at 10:17 AM Ashish Soni <asoni.le...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I made some progress but now i am stuck at this point , Please help >>>>>>>> as looks like i am close to get it working >>>>>>>> >>>>>>>> I have everything running in docker container including mesos slave >>>>>>>> and master >>>>>>>> >>>>>>>> When i try to submit the pi example i get below error >>>>>>>> *Error: Cannot load main class from JAR file:/opt/spark/Example* >>>>>>>> >>>>>>>> Below is the command i use to submit as a docker container >>>>>>>> >>>>>>>> docker run -it --rm -e SPARK_MASTER="mesos://10.0.2.15:7077" -e >>>>>>>> SPARK_IMAGE="spark_driver:latest" spark_driver:latest >>>>>>>> ./bin/spark-submit >>>>>>>> --deploy-mode cluster --name "PI Example" --class >>>>>>>> org.apache.spark.examples.SparkPi --driver-memory 512m >>>>>>>> --executor-memory >>>>>>>> 512m --executor-cores 1 >>>>>>>> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Mar 1, 2016 at 2:59 PM, Timothy Chen <t...@mesosphere.io> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Can you go through the Mesos UI and look at the driver/executor >>>>>>>>> log from steer file and see what the problem is? >>>>>>>>> >>>>>>>>> Tim >>>>>>>>> >>>>>>>>> On Mar 1, 2016, at 8:05 AM, Ashish Soni <asoni.le...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Not sure what is the issue but i am getting below error when i >>>>>>>>> try to run spark PI example >>>>>>>>> >>>>>>>>> Blacklisting Mesos slave value: "5345asdasdasdkas234234asdasdasdasd" >>>>>>>>> due to too many failures; is Spark installed on it? >>>>>>>>> WARN TaskSchedulerImpl: Initial job has not accepted any >>>>>>>>> resources; check your cluster UI to ensure that workers are >>>>>>>>> registered and have sufficient resources >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 29, 2016 at 1:39 PM, Sathish Kumaran Vairavelu < >>>>>>>>> vsathishkuma...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> May be the Mesos executor couldn't find spark image or the >>>>>>>>>> constraints are not satisfied. Check your Mesos UI if you see Spark >>>>>>>>>> application in the Frameworks tab >>>>>>>>>> >>>>>>>>>> On Mon, Feb 29, 2016 at 12:23 PM Ashish Soni < >>>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> What is the Best practice , I have everything running as docker >>>>>>>>>>> container in single host ( mesos and marathon also as docker >>>>>>>>>>> container ) >>>>>>>>>>> and everything comes up fine but when i try to launch the spark >>>>>>>>>>> shell i >>>>>>>>>>> get below error >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> SQL context available as sqlContext. >>>>>>>>>>> >>>>>>>>>>> scala> val data = sc.parallelize(1 to 100) >>>>>>>>>>> data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] >>>>>>>>>>> at parallelize at <console>:27 >>>>>>>>>>> >>>>>>>>>>> scala> data.count >>>>>>>>>>> [Stage 0:> >>>>>>>>>>> (0 + 0) / 2]16/02/29 18:21:12 WARN TaskSchedulerImpl: Initial >>>>>>>>>>> job has >>>>>>>>>>> not accepted any resources; check your cluster UI to ensure that >>>>>>>>>>> workers >>>>>>>>>>> are registered and have sufficient resources >>>>>>>>>>> 16/02/29 18:21:27 WARN TaskSchedulerImpl: Initial job has not >>>>>>>>>>> accepted any resources; check your cluster UI to ensure that >>>>>>>>>>> workers are >>>>>>>>>>> registered and have sufficient resources >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Feb 29, 2016 at 12:04 PM, Tim Chen <t...@mesosphere.io> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> No you don't have to run Mesos in docker containers to run >>>>>>>>>>>> Spark in docker containers. >>>>>>>>>>>> >>>>>>>>>>>> Once you have Mesos cluster running you can then specfiy the >>>>>>>>>>>> Spark configurations in your Spark job (i.e: >>>>>>>>>>>> spark.mesos.executor.docker.image=mesosphere/spark:1.6) >>>>>>>>>>>> and Mesos will automatically launch docker containers for you. >>>>>>>>>>>> >>>>>>>>>>>> Tim >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Feb 29, 2016 at 7:36 AM, Ashish Soni < >>>>>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yes i read that and not much details here. >>>>>>>>>>>>> >>>>>>>>>>>>> Is it true that we need to have spark installed on each mesos >>>>>>>>>>>>> docker container ( master and slave ) ... >>>>>>>>>>>>> >>>>>>>>>>>>> Ashish >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Feb 26, 2016 at 2:14 PM, Tim Chen <t...@mesosphere.io> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html should >>>>>>>>>>>>>> be the best source, what problems were you running into? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Tim >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Feb 26, 2016 at 11:06 AM, Yin Yang < >>>>>>>>>>>>>> yy201...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Have you read this ? >>>>>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Feb 26, 2016 at 11:03 AM, Ashish Soni < >>>>>>>>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi All , >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is there any proper documentation as how to run spark on >>>>>>>>>>>>>>>> mesos , I am trying from the last few days and not able to >>>>>>>>>>>>>>>> make it work. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please help >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >> >
