Hello,
I am running Standalone spark server in EC2. My cluster has 1 master and 16
worker nodes. I have a jenkins server that calls spark-submit command like
this
/mnt/services/spark/bin/spark-submit --master spark://172.22.6.181:6066
--deploy-mode cluster --conf spark.driver.maxResultSize=1g
--executor-cores 4 --total-executor-cores 4 --executor-memory 5g
--driver-memory 1g --class com.test.HelloSpark /opt/helloworld.jar
--verbose
The application executes ok but one problem i am facing is when i go to
Spark Web UI and go to Application: page and then click on `Application
Detail UI`, it doesn't work
What i noticed is when i call the spark-submit from jenkins server the
Application Details UI link points to jenkinsurl:4040 . If i execute the
spark-submit from master node then it points to masternode:4040 and if i
execute it from worker node 1 it takes me to workernode1:4040, instead of
pointing to driver:4040
Please note that i am setting SPARK_PUBLIC_DNS= parameter with value of ec2
public hostname in spark-env.sh on each of these machines.
I was wondering how does spark figure out value of Application Details UI, i
noticed that in SparkUI.scala it gets
protected val publicHostName =
Option(conf.getenv("SPARK_PUBLIC_DNS")).getOrElse(localHostName)
So it should have got the value from spark-env.sh of machine where driver is
running but seems like it always points to value of SPARK_PUBLIC_DNS of the
machine from which i invoked spark-submit.sh
Thanks
Sunil
