Re: Spark on k8s cluster mode, from outside of the cluster.

2021-10-27 Thread Fabrizio Fab
Hi Philipp, okay, I realized just now of my HUGE misunderstanding ! The "double-spark-submit" patter is just the standard spark-on-k8s way of running spark applications in cluster mode: the 1st "spark-submit" in "cluster mode" is started from the client (in the zeppelin host, in our case), then

Re: Spark on k8s cluster mode, from outside of the cluster.

2021-10-27 Thread Philipp Dallig
Hi Fabrizio, We have two connections. First, the Zeppelin interpreter opens a connection to the Zeppelin server to register and to send back the interpreter output. The Zeppelin server is the CALLBACK_HOST and the PORT indicates where the Zeppelin server opened the Thrift service for the Zepp