In my experience, Spark in k8s inherited the pods hostname (of the form podname.namespace) which is not resolvable by default, and so definitely won’t be outside the cluster. I set the driver.host to the IP of the pod, and assuming you can route to the cluster nodes, that should work.
Despite this being spark on k8s, it’s still setting up spark in client mode, so I imagine it would be much the same. Ray Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: [email protected] on behalf of Mariano Simone <[email protected]> Sent: Sunday, February 3, 2019 1:33 am To: Project Jupyter Subject: [jupyter] JupyterHub + Kubernetes + Remote Spark HELP Hello guys, I have a JupyterHub deployed in a Kubernetes cluster. I can spawn notebooks just fine. The problem arise when I try to spawn a spark session and connect to a remote server. I can see the application on the remote spark server but the server can't connect back to the driver. How can I fix this? I tried --net=host on docker but it doesn't work. Any way to get this working? -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To post to this group, send email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/143d0081-294a-4898-b162-2cfd4153de58%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/143d0081-294a-4898-b162-2cfd4153de58%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/SY3PR01MB2185A400315E76B63C5B1328C16C0%40SY3PR01MB2185.ausprd01.prod.outlook.com. For more options, visit https://groups.google.com/d/optout.
