Re: [OMPI users] Docker Cluster Queue Manager

Good morning, Dave, Amongst reasons for not running Docker, a major one that I didn't notice > raised is that containers are not started by the resource manager, but > by a privileged daemon, so the resource manager can't directly control > or monitor them. > There's an endless debate

Re: [OMPI users] Docker Cluster Queue Manager

Rob Nagler writes: > Thanks, John. I sometimes wonder if I'm the only one out there with this > particular problem. > > Ralph, thanks for sticking with me. :) Using a pool of uids doesn't really > work due to the way cgroups/containers works. It also would require > changing the permissions of al

Re: [OMPI users] Docker Cluster Queue Manager

On 06/06/2016 06:32 PM, Rob Nagler wrote: Thanks, John. I sometimes wonder if I'm the only one out there with this particular problem. Ralph, thanks for sticking with me. :) Using a pool of uids doesn'

Re: [OMPI users] Docker Cluster Queue Manager

> On Jun 6, 2016, at 10:18 AM, Rob Nagler wrote: > > Ralph, > FWIW: I haven’t seen it before. > > Good to know. > > > Not sure I understand the issue, but I have no knowledge of Jupyter or why > you are using it. From what I can see, it appears that your choice of tools > may be complica

Re: [OMPI users] Docker Cluster Queue Manager

Ralph, > FWIW: I haven’t seen it before. > Good to know. > > Not sure I understand the issue, but I have no knowledge of Jupyter or why > you are using it. From what I can see, it appears that your choice of tools > may be complicating your solution - I’d suggest perhaps focusing on solving > t

Re: [OMPI users] Docker Cluster Queue Manager

> On Jun 6, 2016, at 8:32 AM, Rob Nagler wrote: > > Thanks, John. I sometimes wonder if I'm the only one out there with this > particular problem. FWIW: I haven’t seen it before. > > Ralph, thanks for sticking with me. :) Using a pool of uids doesn't really > work due to the way cgroups/con

Re: [OMPI users] Docker Cluster Queue Manager

Thanks, John. I sometimes wonder if I'm the only one out there with this particular problem. Ralph, thanks for sticking with me. :) Using a pool of uids doesn't really work due to the way cgroups/containers works. It also would require changing the permissions of all of the user's files, which wou

Re: [OMPI users] Docker Cluster Queue Manager

That's why they have acl in ZoL, no? just bring up a new filesystem for each container, with acl so only the owning container can use that fs, and you should be done, no? To be clear, each container would have to have a unique uid for this to work, but together

Re: [OMPI users] Docker Cluster Queue Manager

Is this really required here ? I was under the impression the web server already run user tasks in a container. all tasks run with the same unix id, but that is fine since isolation is provided by the container. did I get it right ? I was thinking of an other approach, which is run the containers

Re: [OMPI users] Docker Cluster Queue Manager

Thought about this some more, and I wonder if there isn’t a simpler solution: * create a worker pool of userid’s that represents the maximum number of simultaneous users you are willing to support. This could be very large, if you want * when a worker id becomes available, pull the next email f

Re: [OMPI users] Docker Cluster Queue Manager

Rob, I am not familair with wakari.io However what you say about the Unix userid problem is very relevant to many 'shared infrastructure' projects and is a topic which comes up in discussions about them. Teh concern there is, as you say, if the managers of the system have a global filesystem, with

Re: [OMPI users] Docker Cluster Queue Manager

Thanks! SLURM Elastic Computing seems like it might do the trick. I need to try it out. xCAT is interesting, too. It seems to be the HPC version of Salt'ed Cobbler. :) I don't know that it's so important for our problem. We have a small cluster for testing against the cloud, primarily. I could se

Re: [OMPI users] Docker Cluster Queue Manager

For provisioning, I personally use xCAT, which just started supporting docker http://xcat-docs.readthedocs.io/en/stable/advanced/docker/lifecycle_management.html Together with slurm elastic computing feature http://xcat-docs.readthedocs.io/en/stable/advanced/docker/lifecycle_ma

Re: [OMPI users] Docker Cluster Queue Manager

Hi Daniel, Thanks. Shifter is also interesting. However, it assumes our users map to a Unix user id, and therefore the access to the shared file system can be controlled by normal Unix permissions. That's not scalable, and makes for quite a bit of complexity. Each node must know about each user s

Re: [OMPI users] Docker Cluster Queue Manager

Did you check shifter? https://www.nersc.gov/assets/Uploads/cug2015udi.pdf , https://www.nersc.gov/assets/Uploads/cug2015udi.pdf , http://www.nersc.gov/research-and-development/user-defined-images/ , https://github.com/NERSC/shifter On 06/03/2016 01:58 AM, Rob Na

Re: [OMPI users] Docker Cluster Queue Manager

Hi John, > What is the use case here - are you just wanting the codes to execute with one given Unix ID? Are you familiar with wakari.io? That's an example of what we want to do, but with the ability to start jobs. Rescale.com is another example of a web-based job submission mechanism. JupyterHu

Re: [OMPI users] Docker Cluster Queue Manager

Rob, I really think you should look at the FAQ http://singularity.lbl.gov/#faq Also I don;t understand what you mean by 'Out users don't have Unix user IDs' That is no problem of course - I have worked with Centrify and Samba, where you can define mappings between Windows users and Unix IDs or gro

Re: [OMPI users] Docker Cluster Queue Manager

Hi John, Thanks for your thoughts. Lots of new technology out there! have you looked at Singularity > https://github.com/gmkurtzer/singularity/releases/tag/2.0 > Looks very cool, but it doesn't address our problem. We already have the images built with our codes. Our users don't have Unix user i

Re: [OMPI users] Docker Cluster Queue Manager

Rob, have you looked at Singularity https://github.com/gmkurtzer/singularity/releases/tag/2.0 It is a new containerisation framework aimed squarely at HPC. Also you mention Juyputer. I am learning Julia at the moment, and I looked at the parallel facilities yesterday https://github.com/JuliaParal

Re: [OMPI users] Docker Cluster Queue Manager

Thanks, Ralph. I'm not sure I explained the problem clearly. Salt and JupyterHub are distractions, sorry. I have code which "wires up" a cluster for MPI. What I need is scheduler that allows users to: * Select which Docker image they'd like to wire up * Request a number of nodes/cores * Understan

Re: [OMPI users] Docker Cluster Queue Manager

I’m afraid I’m not familiar with JupyterHub at all, or Salt. All you really need is: * a scheduler that understands the need to start all the procs at the same time - i.e., as a block * wireup support for the MPI procs themselves If JupyterHub can do the first, then you could just have it laun

[OMPI users] Docker Cluster Queue Manager

We would like to use MPI on Docker with arbitrarily configured clusters (e.g. created with StarCluster or bare metal). What I'm curious about is if there is a queue manager that understands Docker, file systems, MPI, and OpenAuth. JupyterHub does a lot of this, but it doesn't interface with MPI. Id