I did some further investigation.
If I launch a driver in cluster mode with master IPs like
spark://:7077,:7077, the the driver is launched with
both IPs and -Dspark.master property has both IPs.
But within the logs I see the following, it causes 20 second delay while
launching each driver
18/06/
We switched the port from 7077 to 6066 because we were losing 20 seconds each
time we launched a driver. 10 seconds for failing to submit the driver on
:7077. After losing 20 seconds, it used to fallback to some old way of
driver submitions.
With 6066 we don't lose any time.
--
Sent from: http:
We recently transitioned from client mode to cluster mode with Spark
Standalone deployment. We are using 2.2.1. We are also using SparkLauncher
to launch the driver.
The problem is that when my Driver is launched the spark.master property
(-Dspark.master) is set to only primary master IP. Somethin
