hi, all, I am playing with docker, trying to create a spark cluster with docker containers.
since spark master, worker, driver all need to visit each other, I configured a dns server, and set hostname and domain name of each node. but when spark master start up, it seems to be using hostname without domain name. I see lines in sbin/start-master.sh if [ "$SPARK_MASTER_IP" = "" ]; then SPARK_MASTER_IP=`hostname` fi and when trying to use spark-shell, it seems also listen to hostname, and worker failed to connect to driver by hostname. I manually set export SPARK_MASTER_IP=my.master.domain.name export SPARK_LOCAL_IP=my.master.domain.name everything works fine. so I am thinking where can I change this, I found SPARK_PUBLIC_DNS in spark configuration, but document is very simple, I don't know what it do, and what value should I set. should I use this setting and how? thanks
