Hello Kevin, Thanks for taking a moment to reply.
Yes, you are correct, my spark cluster is remote. I’m deploying in client mode. The only problem I can not manage to fix is that the applications spawned on the remote spark cluster can’t connect back to the driver inside the notebook. I tried adding net=host in hub.extraConfig but I don’t think that works. On Sun, 3 Feb 2019 at 16:20 Kevin Bates <[email protected]> wrote: > Hi Mariano, > > Per Luciano's response to the "Docker jupyter kernel" thread you may want > to check out Enterprise Gateway <https://jupyter.org/enterprise_gateway/>. > When Notebook is configured to point to EG > <https://blog.jupyter.org/on-demand-notebooks-with-jupyterhub-jupyter-enterprise-gateway-and-kubernetes-e8e423695cbf> > via the NB2KG server extension, the kernel management is proxied to EG. In > Kubernetes, EG launches kernels in their own pods across the cluster. We > provide kernel images <https://hub.docker.com/u/elyra> configured with > Spark 2.4 where EG uses spark-submit cluster mode to launch the kernel. > The kernel pod is the spark driver and spark is executed within the k8s > cluster using K8s as the resource manager. > > Since it sounds like your Spark cluster is external to Kubernetes and > assuming you can't use Spark on K8s, there are a couple more options you > could take via EG. You could do something similar to the YARN-based > kernelspecs > <https://github.com/jupyter/enterprise_gateway/tree/master/etc/kernelspecs> > we provide. In this case, the kernel would be launched in cluster mode - > so it's running as the spark driver in the remote cluster. Or you may want > to take a look at using our spark-based kernel images but launched as > regular kernels (as opposed to the spark-submit launch). You can then > either create the spark context from within the notebook cell (as you're > probably doing) or convey the necessary information to the pod's launch to > have the image's startup script create the spark context. In this case, > the kernel would be the Spark driver running in client mode. In any case, > we'd be happy to work with you. I agree that you're likely running into a > container network issue. > > Best regards, > Kevin. > > On Saturday, February 2, 2019 at 6:33:01 AM UTC-8, Mariano Simone wrote: >> >> Hello guys, >> >> I have a JupyterHub deployed in a Kubernetes cluster. >> >> I can spawn notebooks just fine. The problem arise when I try to spawn a >> spark session and connect to a remote server. >> >> I can see the application on the remote spark server but the server can't >> connect back to the driver. >> >> How can I fix this? I tried --net=host on docker but it doesn't work. >> >> Any way to get this working? >> > -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jupyter/4452688c-f46f-46df-8bd7-b01357bf601e%40googlegroups.com > <https://groups.google.com/d/msgid/jupyter/4452688c-f46f-46df-8bd7-b01357bf601e%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAFGcxPmED5-BRyR3jpyTu%3DQLqBPjGg%3D0qB1hCO1AUpCCPqLxvQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
