When you say driver running on mesos can you explain how are you doing that...??
> On Mar 10, 2016, at 4:44 PM, Eran Chinthaka Withana
> wrote:
>
> Yanling I'm already running the driver on mesos (through docker). FYI, I'm
> running this on cluster mode with MesosClusterDispatcher.
>
> Mac (c
Yanling I'm already running the driver on mesos (through docker). FYI, I'm
running this on cluster mode with MesosClusterDispatcher.
Mac (client) > MesosClusterDispatcher > Driver running on Mesos -->
Workers running on Mesos
My next step is to run MesosClusterDispatcher in mesos through
Hi Everyone, a quick question with in this context. What is the underneath
persistent storage that you guys are using? With regards to this
containerized environment? Thanks
On Thursday, March 10, 2016, yanlin wang wrote:
> How you guys make driver docker within container to be reachable from
>
How you guys make driver docker within container to be reachable from spark
worker ?
Would you share your driver docker? i am trying to put only driver in docker
and spark running with yarn outside of container and i don’t want to use
—net=host
Thx
Yanlin
> On Mar 10, 2016, at 11:06 AM, Gui
Glad to hear it. Thanks all for sharing your solutions.
Le jeu. 10 mars 2016 19:19, Eran Chinthaka Withana
a écrit :
> Phew, it worked. All I had to do was to add *export
> SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6"
> *before calling spark-sub
Phew, it worked. All I had to do was to add *export
SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6"
*before calling spark-submit. Guillaume, thanks for the pointer.
Timothy, thanks for looking into this. Looking forward to see a fix soon.
Thanks,
Eran
Hi Timothy
What version of spark are you guys running?
>
I'm using Spark 1.6.0. You can see the Dockerfile I used here:
https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
> And also did you set the working dir in your image to be spark home?
>
Yes I did
Hi Eran,
I need to investigate but perhaps that's true, we're using SPARK_JAVA_OPTS
to pass all the options and not --conf.
I'll take a look at the bug, but if you can try the workaround and see if
that fixes your problem.
Tim
On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <
eran.chin
t the
>>>> executors
>>>> fail:
>>>> "sh: /some/spark/home/bin/spark-class: No such file or directory"
>>>>
>>>> Looking on MESOS slaves log, I think that the executors do not run
>>>> inside
>>>> docker: &
s.
>>>
>>> *It seems that the spark conf that I gave in the first spark-submit is
>>> not
>>> transmitted to the Driver submitted conf*, when launched in the docker
>>> container. The only workaround I f
s log, I think that the executors do not run inside
>>>> docker: "docker.cpp:775] No container info found, skipping launch". As my
>>>> Mesos slaves do not have spark installed, it fails.
>>>>
>>>> *It seems that the spark conf that I gave
os slaves do not have spark installed, it fails.
>>>
>>> *It seems that the spark conf that I gave in the first spark-submit is not
>>> transmitted to the Driver submitted conf*, when launched in the docker
>>> container. The only workaround I found is to modif
to the Driver submitted conf*, when launched in the docker
>> container. The only workaround I found is to modify my Docker image in
>> order
>> to define inside its spark conf the spark.mesos.executor.docker.image
>> property. This way, my executors get the conf well and are launch
conf well and are launched inside
> docker on Mesos. This seems a little complicated to me, and I feel the
> configuration passed to the early spark-submit should be transmitted to the
> Driver submit...
>
>
>
> --
> View this message in context:
> http://apache-spark-use
ation passed to the early spark-submit should be transmitted to the
Driver submit...
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html
Sent from the Apache Spark User List mailing l
15 matches
Mail list logo
