The idea of splitting out the kernels into seperate container just makes a lot of sense to me. I’ve used enterprise gateway and it works pretty well.
However, we had one big gotcha with this approach; the filesystem for the kernel is not the same as in Jupiter - our users are used to being able to reference other files they upload/see in the Jupiter UI. I tried a few approaches to this (mounting same home directory into kernel, copying files to kernel) but they all resulted in more complexity and there were consistency issues. Given I was running everything in k8s and that the kernels shares so much with the notebook image, it just seemed easier to keep them in one image for now. I’d be very curious to see how others solved or circumvented this. Ray Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: [email protected] on behalf of Luciano Resende <[email protected]> Sent: Saturday, February 2, 2019 10:37 am To: [email protected] Subject: Re: [jupyter] Docker jupyter kernel Have a look at Jupyter Enterprise Gateway, which enables and manage remote kernels in multiple cluster environments including container based ones such: Docker, Swarm, Kubernetes, etc https://jupyter.org/enterprise_gateway/ https://github.com/jupyter/enterprise_gateway Please let us know if you have specific questions after checking it out. On Fri, Feb 1, 2019 at 10:59 AM Matt Morgis <[email protected]> wrote: > > For anyone in 2019 looking to do this, we built a prototype here: > https://github.com/tamera-lanham/ipython-kernel-docker > > Similar to the approach in the Gist, except instead of a Python file running > the container, we tell Jupyter to do it instead. > > On Tuesday, July 10, 2018 at 4:01:53 AM UTC-4, Ken Jiiii wrote: >> >> Hi guys, >> >> I think I have the same use case and I was wondering whether this discussion >> is still up to date. >> The idea is to have a Jupyter running on a local machine which has one or >> more docker containers running at the same time. These containers provide >> for example different python versions like 3.6 and 3.7. >> Now the question is how to add an external kernel to Jupyter. The kernel is >> of course running in the docker container. >> >> https://gist.github.com/mariusvniekerk/09062bc8974e5d1f4af6 is this approach >> still valid @ Marius? >> >> I have also read that it is possible to connect via ssh to a remote kernel >> in Jupyter but in that case SSH needs to be configured in the container. >> >> Can anybody tell me what solution is still working for him? >> >> Thanks a lot in advance and kind regards! >> >> Am Mittwoch, 26. Juli 2017 14:23:08 UTC+2 schrieb Ashwin Srinath: >>> >>> We have used Singularity (http://singularity.lbl.gov/) containers in >>> Jupyter Notebooks with relative ease. Some notes available here: >>> >>> https://github.com/clemsonciti/singularity-in-jupyter-notebook >>> >>> Thanks, >>> Ashwin >>> >>> On Tue, Jul 25, 2017 at 5:10 AM, Stojan Jovanović wrote: >>>> >>>> Hi James, >>>> >>>> I'm currently buidling something very similar to what you're talking about. >>>> >>>> I've currently got it set up so that I can access multiple Dockers, >>>> containing isolated machine learning models, through a Jupyter notebook >>>> (located in a third Docker), via SSH. >>>> >>>> It wasn't super difficult to do, although I'm not claiming it was done >>>> very elegantly. >>>> >>>> If you're interested, you can take a look here >>>> https://github.com/stojan211287/DockerSSH. I've uploaded a minimal >>>> example, consisting of one "drone" and one "overlord" container. The >>>> overlord issues commands via SSH, the drone complies and delivers. >>>> >>>> As it stands now, I've based the images on Alpine 3.6 and am currently >>>> using them as base images for further development.- the overlord get >>>> Jupyter installed on top of it, and the drone, for example, can host >>>> scikit-learn. >>>> >>>> On Wednesday, 24 May 2017 23:10:34 UTC+2, James wrote: >>>>> >>>>> Hey, sorry to revive this thread again, but having docker container >>>>> kernels (and not whole jupyter server systems) would be very useful for >>>>> me. My use case is having certain hard to build scientific software >>>>> installed within the container. That way you could call out to them using >>>>> python's subprocess calls from within the notebook. My goal would be to >>>>> make several kernels, accessible from the same notebook server, to act as >>>>> a toolkit of sorts for my lab. Ideally having the kernels in containers >>>>> would make them easy to share and install in sister labs at other >>>>> institutions for use in their Jupyter ecosystem. Thank you for any >>>>> guidance! >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Project Jupyter" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jupyter/b22a6d05-25c6-4e24-ae2b-df9ea723ed6c%40googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> > -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jupyter/36cc59a4-c95e-4557-a34a-4ab928b85be3%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/ -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAGU5speB-aPRRr3uWpmCj6%3DnOFioeEG9F%3DR-bfPm-Us_aj6%2BxQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/SY3PR01MB218593599D6C59E1CC2A1CDEC16C0%40SY3PR01MB2185.ausprd01.prod.outlook.com. For more options, visit https://groups.google.com/d/optout.
