I have no luck and i would to ask the question to spark committers will this be ever designed to run on mesos ?
spark app as a docker container not working at all on mesos ,if any one would like the code i can send it over to have a look. Ashish On Wed, Mar 2, 2016 at 12:23 PM, Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Try passing jar using --jars option > > On Wed, Mar 2, 2016 at 10:17 AM Ashish Soni <asoni.le...@gmail.com> wrote: > >> I made some progress but now i am stuck at this point , Please help as >> looks like i am close to get it working >> >> I have everything running in docker container including mesos slave and >> master >> >> When i try to submit the pi example i get below error >> *Error: Cannot load main class from JAR file:/opt/spark/Example* >> >> Below is the command i use to submit as a docker container >> >> docker run -it --rm -e SPARK_MASTER="mesos://10.0.2.15:7077" -e >> SPARK_IMAGE="spark_driver:latest" spark_driver:latest ./bin/spark-submit >> --deploy-mode cluster --name "PI Example" --class >> org.apache.spark.examples.SparkPi --driver-memory 512m --executor-memory >> 512m --executor-cores 1 >> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >> >> >> On Tue, Mar 1, 2016 at 2:59 PM, Timothy Chen <t...@mesosphere.io> wrote: >> >>> Can you go through the Mesos UI and look at the driver/executor log from >>> steer file and see what the problem is? >>> >>> Tim >>> >>> On Mar 1, 2016, at 8:05 AM, Ashish Soni <asoni.le...@gmail.com> wrote: >>> >>> Not sure what is the issue but i am getting below error when i try to >>> run spark PI example >>> >>> Blacklisting Mesos slave value: "5345asdasdasdkas234234asdasdasdasd" >>> due to too many failures; is Spark installed on it? >>> WARN TaskSchedulerImpl: Initial job has not accepted any resources; >>> check your cluster UI to ensure that workers are registered and have >>> sufficient resources >>> >>> >>> On Mon, Feb 29, 2016 at 1:39 PM, Sathish Kumaran Vairavelu < >>> vsathishkuma...@gmail.com> wrote: >>> >>>> May be the Mesos executor couldn't find spark image or the constraints >>>> are not satisfied. Check your Mesos UI if you see Spark application in the >>>> Frameworks tab >>>> >>>> On Mon, Feb 29, 2016 at 12:23 PM Ashish Soni <asoni.le...@gmail.com> >>>> wrote: >>>> >>>>> What is the Best practice , I have everything running as docker >>>>> container in single host ( mesos and marathon also as docker container ) >>>>> and everything comes up fine but when i try to launch the spark shell i >>>>> get below error >>>>> >>>>> >>>>> SQL context available as sqlContext. >>>>> >>>>> scala> val data = sc.parallelize(1 to 100) >>>>> data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at >>>>> parallelize at <console>:27 >>>>> >>>>> scala> data.count >>>>> [Stage 0:> (0 >>>>> + 0) / 2]16/02/29 18:21:12 WARN TaskSchedulerImpl: Initial job has not >>>>> accepted any resources; check your cluster UI to ensure that workers are >>>>> registered and have sufficient resources >>>>> 16/02/29 18:21:27 WARN TaskSchedulerImpl: Initial job has not accepted >>>>> any resources; check your cluster UI to ensure that workers are registered >>>>> and have sufficient resources >>>>> >>>>> >>>>> >>>>> On Mon, Feb 29, 2016 at 12:04 PM, Tim Chen <t...@mesosphere.io> wrote: >>>>> >>>>>> No you don't have to run Mesos in docker containers to run Spark in >>>>>> docker containers. >>>>>> >>>>>> Once you have Mesos cluster running you can then specfiy the Spark >>>>>> configurations in your Spark job (i.e: >>>>>> spark.mesos.executor.docker.image=mesosphere/spark:1.6) >>>>>> and Mesos will automatically launch docker containers for you. >>>>>> >>>>>> Tim >>>>>> >>>>>> On Mon, Feb 29, 2016 at 7:36 AM, Ashish Soni <asoni.le...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Yes i read that and not much details here. >>>>>>> >>>>>>> Is it true that we need to have spark installed on each mesos docker >>>>>>> container ( master and slave ) ... >>>>>>> >>>>>>> Ashish >>>>>>> >>>>>>> On Fri, Feb 26, 2016 at 2:14 PM, Tim Chen <t...@mesosphere.io> wrote: >>>>>>> >>>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html should >>>>>>>> be the best source, what problems were you running into? >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> On Fri, Feb 26, 2016 at 11:06 AM, Yin Yang <yy201...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Have you read this ? >>>>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html >>>>>>>>> >>>>>>>>> On Fri, Feb 26, 2016 at 11:03 AM, Ashish Soni < >>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi All , >>>>>>>>>> >>>>>>>>>> Is there any proper documentation as how to run spark on mesos , >>>>>>>>>> I am trying from the last few days and not able to make it work. >>>>>>>>>> >>>>>>>>>> Please help >>>>>>>>>> >>>>>>>>>> Ashish >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >>
