Hi Mariano, Per Luciano's response to the "Docker jupyter kernel" thread you may want to check out Enterprise Gateway <https://jupyter.org/enterprise_gateway/>. When Notebook is configured to point to EG <https://blog.jupyter.org/on-demand-notebooks-with-jupyterhub-jupyter-enterprise-gateway-and-kubernetes-e8e423695cbf> via the NB2KG server extension, the kernel management is proxied to EG. In Kubernetes, EG launches kernels in their own pods across the cluster. We provide kernel images <https://hub.docker.com/u/elyra> configured with Spark 2.4 where EG uses spark-submit cluster mode to launch the kernel. The kernel pod is the spark driver and spark is executed within the k8s cluster using K8s as the resource manager.
Since it sounds like your Spark cluster is external to Kubernetes and assuming you can't use Spark on K8s, there are a couple more options you could take via EG. You could do something similar to the YARN-based kernelspecs <https://github.com/jupyter/enterprise_gateway/tree/master/etc/kernelspecs> we provide. In this case, the kernel would be launched in cluster mode - so it's running as the spark driver in the remote cluster. Or you may want to take a look at using our spark-based kernel images but launched as regular kernels (as opposed to the spark-submit launch). You can then either create the spark context from within the notebook cell (as you're probably doing) or convey the necessary information to the pod's launch to have the image's startup script create the spark context. In this case, the kernel would be the Spark driver running in client mode. In any case, we'd be happy to work with you. I agree that you're likely running into a container network issue. Best regards, Kevin. On Saturday, February 2, 2019 at 6:33:01 AM UTC-8, Mariano Simone wrote: > > Hello guys, > > I have a JupyterHub deployed in a Kubernetes cluster. > > I can spawn notebooks just fine. The problem arise when I try to spawn a > spark session and connect to a remote server. > > I can see the application on the remote spark server but the server can't > connect back to the driver. > > How can I fix this? I tried --net=host on docker but it doesn't work. > > Any way to get this working? > -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/4452688c-f46f-46df-8bd7-b01357bf601e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
