Re: [OMPI users] OpenMPI and Singularity

2017-02-20 Thread Bennet Fauber
Ralph, attached please find a file with the results of $ mpirun -d singularity exec ./mpi_test.img /usr/bin/ring I pressed Ctrl-C at about line 200 of the output file. I hope there is something useful in it. My nodefile looks like this nyx6219 nyx6219 nyx6145 nyx6145 nyx6191 nyx6191 nyx615

Re: [OMPI users] OpenMPI and Singularity

2017-02-20 Thread r...@open-mpi.org
If you can send us some more info on how it breaks, that would be helpful. I’ll file it as an issue so we can track things Thanks Ralph > On Feb 20, 2017, at 9:13 AM, Bennet Fauber wrote: > > I got mixed results when bringing a container that doesn't have the IB > and Torque libraries compile

Re: [OMPI users] OpenMPI and Singularity

2017-02-20 Thread Bennet Fauber
I got mixed results when bringing a container that doesn't have the IB and Torque libraries compiled into the OMPI inside the container to a cluster where it does. The short summary is that mutlinode communication seems unreliable. I can mostly get up to 8 procs, two-per-node, to run, but beyond

Re: [OMPI users] OpenMPI and Singularity

2017-02-17 Thread r...@open-mpi.org
I -think- that is correct, but you may need the verbs library as well - I honestly don’t remember if the configury checks for functions in the library or not. If so, then you’ll need that wherever you build OMPI, but everything else is accurate Good luck - and let us know how it goes! Ralph >

Re: [OMPI users] OpenMPI and Singularity

2017-02-17 Thread Bennet Fauber
Ralph. I will be building from the Master branch at github.com for testing purposes. We are not 'supporting' Singularity container creation, but we do hope to be able to offer some guidance, so I think we can finesse the PMIx version, yes? That is good to know about the verbs headers being the o

Re: [OMPI users] OpenMPI and Singularity

2017-02-17 Thread r...@open-mpi.org
The embedded Singularity support hasn’t made it into the OMPI 2.x release series yet, though OMPI will still work within a Singularity container anyway. Compatibility across the container boundary is always a problem, as your examples illustrate. If the system is using one OMPI version and the c

[OMPI users] OpenMPI and Singularity

2017-02-17 Thread Bennet Fauber
I am wishing to follow the instructions on the Singularity web site, http://singularity.lbl.gov/docs-hpc to test Singularity and OMPI on our cluster. My previously normal configure for the 1.x series looked like this. ./configure --prefix=/usr/local \ --mandir=${PREFIX}/share/man \ -