Hi Spark folks, So on our production Spark cluster, it lives in the data center and I need to attach to a VPN from my laptop, so that I can then submit a Spark application job to the Spark Master (behind the VPN).
However, the problem arises that I have a local IP address on the laptop which is on a separate network segment than the VPN. I figured out that I can set the SparkConf configuration property called "spark.driver.host" to be my VPN IP address - this got me further, because I could initiate a connection to the master. However, it was still not working. I was hoping to set the SparkConf variables "spark.fileserver.uri" and "spark.httpBroadcast.uri" to NOT be my local IP addresses, but the VPN IP addresses, because I think these configuration variables actually pass my application files to the master (in this case, it is a python script). So my spark.fileserver.uri and the Spark Master need to be on the same network segment (the VPN subnetwork ). Am I on the right track? How can I set "spark.fileserver.uri" and "spark.httpBroadcast.uri" ? I see that these are actually run by Jetty server...any thoughts? Thank you so much! -- Άρης Βλασακάκης Aris Vlasakakis
