Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread tmishima
Oscar, As I mentioned before, I've never used SGE. So please ask for Reuti's advise. Only thing I can tell is that you have to use the openmpi 1.8 series to use -map-by slot:pe=N option. Tetsuya > Hi > > Well, with qconf -sq one.q I got the following: > > [oscar@aguia free-noise]$ qconf -sq one

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread tmishima
Reuti, Sorry for confusing you. Under the managed condition, actually -np option is not necessary. So, this cmd line also works for me with Torque. $ qsub -l nodes=10:ppn=N $ mpirun -map-by slot:pe=N ./inverse.exe At least, Ralph confirmed it worked with Slurm and I comfirmed with Torque as show

Re: [OMPI users] Clarification about OpenMPI, slurm and PMI interface

2014-08-20 Thread Ralph Castain
Or you can add -nolocal|--nolocalDo not run any MPI applications on the local node to your mpirun command line and we won't run any application procs on the node where mpirun is executing On Aug 20, 2014, at 4:28 PM, Joshua Ladd wrote: > Hi, Filippo > > When launching with mpirun in

Re: [OMPI users] Clarification about OpenMPI, slurm and PMI interface

2014-08-20 Thread Joshua Ladd
Hi, Filippo When launching with mpirun in a SLURM environment, srun is only being used to launch the ORTE daemons (orteds.) Since the daemon will already exist on the node from which you invoked mpirun, this node will not be included in the list of nodes. SLURM's PMI library is not involved (that

[OMPI users] Clarification about OpenMPI, slurm and PMI interface

2014-08-20 Thread Filippo Spiga
Dear Open MPI experts, I have a problem that is related to the integration of OpenMPI, slurm and PMI interface. I spent some time today with a colleague of mine trying to figure out why we were not able to obtain all H5 profile files (generated by acct_gather_profile) using Open MPI. When I say

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Ralph Castain
On Aug 20, 2014, at 11:16 AM, Reuti wrote: > Am 20.08.2014 um 19:05 schrieb Ralph Castain: > >>> >>> Aha, this is quite interesting - how do you do this: scanning the >>> /proc//status or alike? What happens if you don't find enough free >>> cores as they are used up by other applications al

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Reuti
Am 20.08.2014 um 19:05 schrieb Ralph Castain: >> >> Aha, this is quite interesting - how do you do this: scanning the >> /proc//status or alike? What happens if you don't find enough free >> cores as they are used up by other applications already? >> > > Remember, when you use mpirun to launc

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Oscar Mojica
Hi Well, with qconf -sq one.q I got the following: [oscar@aguia free-noise]$ qconf -sq one.q qname one.q hostlist compute-1-30.local compute-1-2.local compute-1-3.local \ compute-1-4.local compute-1-5.local compute-1-6.local \

Re: [OMPI users] ORTE daemon has unexpectedly failed after launch

2014-08-20 Thread Ralph Castain
yes, i know - it is cmr'd On Aug 20, 2014, at 10:26 AM, Mike Dubman wrote: > btw, we get same error in v1.8 branch as well. > > > On Wed, Aug 20, 2014 at 8:06 PM, Ralph Castain wrote: > It was not yet fixed - but should be now. > > On Aug 20, 2014, at 6:39 AM, Timur Ismagilov wrote: > >> H

Re: [OMPI users] ORTE daemon has unexpectedly failed after launch

2014-08-20 Thread Mike Dubman
btw, we get same error in v1.8 branch as well. On Wed, Aug 20, 2014 at 8:06 PM, Ralph Castain wrote: > It was not yet fixed - but should be now. > > On Aug 20, 2014, at 6:39 AM, Timur Ismagilov wrote: > > Hello! > > As i can see, the bug is fixed, but in Open MPI v1.9a1r32516 i still have > t

Re: [OMPI users] ORTE daemon has unexpectedly failed after launch

2014-08-20 Thread Ralph Castain
It was not yet fixed - but should be now. On Aug 20, 2014, at 6:39 AM, Timur Ismagilov wrote: > Hello! > > As i can see, the bug is fixed, but in Open MPI v1.9a1r32516 i still have > the problem > > a) > $ mpirun -np 1 ./hello_c > > -

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Ralph Castain
On Aug 20, 2014, at 9:04 AM, Reuti wrote: > Am 20.08.2014 um 16:26 schrieb Ralph Castain: > >> On Aug 20, 2014, at 6:58 AM, Reuti wrote: >> >>> Hi, >>> >>> Am 20.08.2014 um 13:26 schrieb tmish...@jcity.maeda.co.jp: >>> Reuti, If you want to allocate 10 procs with N threads,

Re: [OMPI users] No log_num_mtt in Ubuntu 14.04

2014-08-20 Thread Rio Yokota
Dear Roland, Thank you so much. This was very helpful. Best, Rio >> "Mike" == Mike Dubman writes: > >Mike> so, it seems you have old ofed w/o this parameter. Can you >Mike> install latest Mellanox ofed? or check which community ofed >Mike> has it? > > Rio is using the kernel.

Re: [OMPI users] No log_num_mtt in Ubuntu 14.04

2014-08-20 Thread rf
> "Mike" == Mike Dubman writes: Mike> so, it seems you have old ofed w/o this parameter. Can you Mike> install latest Mellanox ofed? or check which community ofed Mike> has it? Rio is using the kernel.org drivers that are part of Ubuntu/3.13.x and log_num_mtt is not a parameter

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Reuti
Am 20.08.2014 um 16:26 schrieb Ralph Castain: > On Aug 20, 2014, at 6:58 AM, Reuti wrote: > >> Hi, >> >> Am 20.08.2014 um 13:26 schrieb tmish...@jcity.maeda.co.jp: >> >>> Reuti, >>> >>> If you want to allocate 10 procs with N threads, the Torque >>> script below should work for you: >>> >>>

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Ralph Castain
On Aug 20, 2014, at 6:58 AM, Reuti wrote: > Hi, > > Am 20.08.2014 um 13:26 schrieb tmish...@jcity.maeda.co.jp: > >> Reuti, >> >> If you want to allocate 10 procs with N threads, the Torque >> script below should work for you: >> >> qsub -l nodes=10:ppn=N >> mpirun -map-by slot:pe=N -np 10 -x

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Reuti
Hi, Am 20.08.2014 um 13:26 schrieb tmish...@jcity.maeda.co.jp: > Reuti, > > If you want to allocate 10 procs with N threads, the Torque > script below should work for you: > > qsub -l nodes=10:ppn=N > mpirun -map-by slot:pe=N -np 10 -x OMP_NUM_THREADS=N ./inverse.exe I played around with givin

Re: [OMPI users] ORTE daemon has unexpectedly failed after launch

2014-08-20 Thread Timur Ismagilov
Hello! As i can see, the bug is fixed, but in Open MPI v1.9a1r32516  i still have the problem a) $ mpirun  -np 1 ./hello_c -- An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun. This

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Ralph Castain
Just to clarify: OMPI will bind the process to *all* N cores, not just to one. On Aug 20, 2014, at 4:26 AM, tmish...@jcity.maeda.co.jp wrote: > Reuti, > > If you want to allocate 10 procs with N threads, the Torque > script below should work for you: > > qsub -l nodes=10:ppn=N > mpirun -map-by

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread tmishima
Reuti, If you want to allocate 10 procs with N threads, the Torque script below should work for you: qsub -l nodes=10:ppn=N mpirun -map-by slot:pe=N -np 10 -x OMP_NUM_THREADS=N ./inverse.exe Then, the openmpi automatically reduces the logical slot count to 10 by dividing real slot count 10N by b

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Reuti
Hi, Am 20.08.2014 um 06:26 schrieb Tetsuya Mishima: > Reuti and Oscar, > > I'm a Torque user and I myself have never used SGE, so I hesitated to join > the discussion. > > From my experience with the Torque, the openmpi 1.8 series has already > resolved the issue you pointed out in combining

[OMPI users] Does multiple Irecv means concurrent receiving ?

2014-08-20 Thread Zhang,Lei(Ecom)
I have a performance problem with receiving. In a single master thread, I made several Irecv calls: Irecv(buf1, ..., tag, ANY_SOURCE, COMM_WORLD) Irecv(buf2, ..., tag, ANY_SOURCE, COMM_WORLD) ... Irecv(bufn, ..., tag, ANY_SOURCE, COMM_WORLD) all of which try to receive from any node for messages

Re: [OMPI users] Running a hybrid MPI+openMP program

2014-08-20 Thread Tetsuya Mishima
Reuti and Oscar, I'm a Torque user and I myself have never used SGE, so I hesitated to join the discussion. >From my experience with the Torque, the openmpi 1.8 series has already resolved the issue you pointed out in combining MPI with OpenMP. Please try to add --map-by slot:pe=8 option, if