My host IP is 192.168.1.16
The VM IP is 10.0.2.15 (using Docker4Mac)
Indeed I tried many things:
1) using the host network (--net=host) but then I cannot access Zeppelin
(localhost:8080 or 127.0.0.1:8080). Zeppelin listens to the address 0.0.0.0
by default (config in zeppelin-site.xml). Changing it to 127.0.0.1 does not
help
2) using bridge network but configure port
- run the container with -p 8080:8080 -p 8081:8081 -p 4040:4040
this time I can access Zeppelin Web UI but then the Spark job hangs
- edit the $ZEPPELIN_HOME/bin/interpreter.sh to start Zeppelin in
"cluster" mode (--deploy-mode cluster) but it does not work
- force the driver host to 192.168.1.16 and driver port to a fix port
(9991) instead of a random port --> the job is launched successfully but
when calling rdd.collect() the worker running on the host machine cannot
send back the result to the driver program inside the container :
6/08/05 14:16:40 ERROR RetryingBlockFetcher: Exception while beginning
fetch of 1 outstanding blocks java.io.IOException: Failed to connect to /
10.0.2.15:42418 at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:193)
The driver program is still listening to the internal IP address on a
dynamic port ....
On Fri, Aug 5, 2016 at 11:41 PM, Trevor Grant <trevor.d.gr...@gmail.com>
wrote:
> Was having some issues with this too.
>
> Did you try using the %sh to ping the machine with spark? e.g. check for
> networking issue.
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> *"Fortunate is he, who is able to know the causes of things." -Virgil*
>
>
> On Fri, Aug 5, 2016 at 3:29 PM, Luciano Resende <luckbr1...@gmail.com>
> wrote:
>
>>
>>
>> On Fri, Aug 5, 2016 at 11:18 PM, Luciano Resende <luckbr1...@gmail.com>
>> wrote:
>>
>>> Not exactly what you want, but I have an example here :
>>> https://github.com/lresende/docker-systemml-notebook
>>>
>>> You should be able to accomplish what you want playing with --link which
>>> I did in the example below (but just with Yarn and HDFS)
>>> https://github.com/lresende/docker-yarn-cluster
>>>
>>>
>> BTW, you might have to use Levy to access the remote Spark.
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>>
>
>
