Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-28 Thread Mark Santcroos
Hi Brock, Angel, Reuti, You might want to look at a tool we developed: http://radical-cybertools.github.io/radical-pilot/index.html This was actually one of the drivers for isolating the persistent ORTE DVM thats being discussed in this thread. With RADICAL-Pilot you can use a Python API to l

Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-27 Thread r...@open-mpi.org
> On Feb 27, 2017, at 9:39 AM, Reuti wrote: > > >> Am 27.02.2017 um 18:24 schrieb Angel de Vicente : >> >> […] >> >> For a small group of users if the DVM can run with my user and there is >> no restriction on who can use it or if I somehow can authorize others to >> use it (via an authority

Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-27 Thread Angel de Vicente
Hi, Reuti writes: > At first I thought you want to run a queuing system inside a queuing > system, but this looks like you want to replace the resource manager. yes, if this could work reasonably well, we could do without the resource manager. > Under which user account the DVM daemons will run

Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-27 Thread Reuti
> Am 27.02.2017 um 18:24 schrieb Angel de Vicente : > > […] > > For a small group of users if the DVM can run with my user and there is > no restriction on who can use it or if I somehow can authorize others to > use it (via an authority file or similar) that should be enough. AFAICS there is n

Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-27 Thread Reuti
Hi, > Am 27.02.2017 um 14:33 schrieb Angel de Vicente : > > Hi, > > "r...@open-mpi.org" writes: >>> With the DVM, is it possible to keep these jobs in some sort of queue, >>> so that they will be executed when the cores get free? >> >> It wouldn’t be hard to do so - as long as it was just a si

Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-27 Thread Angel de Vicente
Hi, "r...@open-mpi.org" writes: >> With the DVM, is it possible to keep these jobs in some sort of queue, >> so that they will be executed when the cores get free? > > It wouldn’t be hard to do so - as long as it was just a simple FIFO > scheduler. I wouldn’t want it to get too complex. a simpl

Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-27 Thread r...@open-mpi.org
> On Feb 27, 2017, at 4:58 AM, Angel de Vicente wrote: > > Hi, > > "r...@open-mpi.org" writes: >> You might want to try using the DVM (distributed virtual machine) >> mode in ORTE. You can start it on an allocation using the “orte-dvm” >> cmd, and then submit jobs to it with “mpirun --hnp ”, w

Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-27 Thread Angel de Vicente
Hi, "r...@open-mpi.org" writes: > You might want to try using the DVM (distributed virtual machine) > mode in ORTE. You can start it on an allocation using the “orte-dvm” > cmd, and then submit jobs to it with “mpirun --hnp ”, where foo > is either the contact info printed out by orte-dvm, or the

Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-23 Thread r...@open-mpi.org
You might want to try using the DVM (distributed virtual machine) mode in ORTE. You can start it on an allocation using the “orte-dvm” cmd, and then submit jobs to it with “mpirun --hnp ”, where foo is either the contact info printed out by orte-dvm, or the name of the file you told orte-dvm to

[OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

2017-02-23 Thread Brock Palen
Is it possible to use mpirun / orte as a load balancer for running serial jobs in parallel similar to GNU Parallel? https://www.biostars.org/p/63816/ Reason is on any major HPC system you normally want to use a resource manager launcher (TM, slurm etc) and not ssh like gnu parallel. I recall the