Re: [OMPI users] Unable to mpirun from within torque

Wed, 07 Sep 2016 06:28:01 -0700 

Hi Gilles,
Thanks for the hint with the machinefile. I know it is not equivalent and i do not intend to use that approach. I just wanted to know whether I could start the program successfully at all. 


Outside torque(4.2), rsh seems to be used which works fine, querying a 
password if no kerberos ticket is there


Here is the output:
[zbh251@a00551 ~]$ mpirun -V
mpirun (Open MPI) 2.0.1
[zbh251@a00551 ~]$ ompi_info | grep ras

                 MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, Component 
v2.0.1)
                 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component 
v2.0.1)
                 MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component 
v2.0.1)

                 MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v2.0.1)

[zbh251@a00551 ~]$ mpirun --mca plm_base_verbose 10 --tag-output 
-display-map hostname
[a00551.science.domain:04104] mca: base: components_register: 
registering framework plm components
[a00551.science.domain:04104] mca: base: components_register: found 
loaded component isolated
[a00551.science.domain:04104] mca: base: components_register: component 
isolated has no register or open function
[a00551.science.domain:04104] mca: base: components_register: found 
loaded component rsh
[a00551.science.domain:04104] mca: base: components_register: component 
rsh register function successful
[a00551.science.domain:04104] mca: base: components_register: found 
loaded component slurm
[a00551.science.domain:04104] mca: base: components_register: component 
slurm register function successful
[a00551.science.domain:04104] mca: base: components_register: found 
loaded component tm
[a00551.science.domain:04104] mca: base: components_register: component 
tm register function successful
[a00551.science.domain:04104] mca: base: components_open: opening plm 
components
[a00551.science.domain:04104] mca: base: components_open: found loaded 
component isolated
[a00551.science.domain:04104] mca: base: components_open: component 
isolated open function successful
[a00551.science.domain:04104] mca: base: components_open: found loaded 
component rsh
[a00551.science.domain:04104] mca: base: components_open: component rsh 
open function successful
[a00551.science.domain:04104] mca: base: components_open: found loaded 
component slurm
[a00551.science.domain:04104] mca: base: components_open: component 
slurm open function successful
[a00551.science.domain:04104] mca: base: components_open: found loaded 
component tm
[a00551.science.domain:04104] mca: base: components_open: component tm 
open function successful
[a00551.science.domain:04104] mca:base:select: Auto-selecting plm 
components
[a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
[isolated]
[a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
[isolated] set priority to 0
[a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
[rsh]
[a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
[rsh] set priority to 10
[a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
[slurm]
[a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
[tm]
[a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
[tm] set priority to 75
[a00551.science.domain:04104] mca:base:select:(  plm) Selected component 
[tm]
[a00551.science.domain:04104] mca: base: close: component isolated 
closed
[a00551.science.domain:04104] mca: base: close: unloading component 
isolated

[a00551.science.domain:04104] mca: base: close: component rsh closed
[a00551.science.domain:04104] mca: base: close: unloading component rsh
[a00551.science.domain:04104] mca: base: close: component slurm closed

[a00551.science.domain:04104] mca: base: close: unloading component 
slurm
[a00551.science.domain:04109] mca: base: components_register: 
registering framework plm components
[a00551.science.domain:04109] mca: base: components_register: found 
loaded component rsh
[a00551.science.domain:04109] mca: base: components_register: component 
rsh register function successful
[a00551.science.domain:04109] mca: base: components_open: opening plm 
components
[a00551.science.domain:04109] mca: base: components_open: found loaded 
component rsh
[a00551.science.domain:04109] mca: base: components_open: component rsh 
open function successful
[a00551.science.domain:04109] mca:base:select: Auto-selecting plm 
components
[a00551.science.domain:04109] mca:base:select:(  plm) Querying component 
[rsh]
[a00551.science.domain:04109] mca:base:select:(  plm) Query of component 
[rsh] set priority to 10
[a00551.science.domain:04109] mca:base:select:(  plm) Selected component 
[rsh]
[a00551.science.domain:04109] [[53688,0],1] bind() failed on error 
Address already in use (98)
[a00551.science.domain:04109] [[53688,0],1] ORTE_ERROR_LOG: Error in 
file oob_usock_component.c at line 228

 Data for JOB [53688,1] offset 0

 ========================   JOB MAP   ========================

 Data for node: a00551  Num slots: 2    Max slots: 0    Num procs: 2

 	Process OMPI jobid: [53688,1] App: 0 Process rank: 0 Bound: socket 
0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket 
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt 
0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt 
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]
 	Process OMPI jobid: [53688,1] App: 0 Process rank: 1 Bound: socket 
1[core 10[hwt 0-1]], socket 1[core 11[hwt 0-1]], socket 1[core 12[hwt 
0-1]], socket 1[core 13[hwt 0-1]], socket 1[core 14[hwt 0-1]], socket 
1[core 15[hwt 0-1]], socket 1[core 16[hwt 0-1]], socket 1[core 17[hwt 
0-1]], socket 1[core 18[hwt 0-1]], socket 1[core 19[hwt 
0-1]]:[../../../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB/BB/BB]



 Data for node: a00553.science.domain	Num slots: 1	Max slots: 0	Num 
procs: 1
 	Process OMPI jobid: [53688,1] App: 0 Process rank: 2 Bound: socket 
0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket 
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt 
0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt 
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]


 =============================================================

[a00551.science.domain:04104] [[53688,0],0] complete_setup on job 
[53688,1]
[a00551.science.domain:04104] [[53688,0],0] plm:base:receive update proc 
state command from [[53688,0],1]
[a00551.science.domain:04104] [[53688,0],0] plm:base:receive got 
update_proc_state for job [53688,1]

[1,0]<stdout>:a00551.science.domain
[1,2]<stdout>:a00551.science.domain

[a00551.science.domain:04104] [[53688,0],0] plm:base:receive update proc 
state command from [[53688,0],1]
[a00551.science.domain:04104] [[53688,0],0] plm:base:receive got 
update_proc_state for job [53688,1]

[1,1]<stdout>:a00551.science.domain
[a00551.science.domain:04109] mca: base: close: component rsh closed
[a00551.science.domain:04109] mca: base: close: unloading component rsh
[a00551.science.domain:04104] mca: base: close: component tm closed
[a00551.science.domain:04104] mca: base: close: unloading component tm

On 2016-09-07 14:41, Gilles Gouaillardet wrote:

Hi,

Which version of Open MPI are you running ?

I noted that though you are asking three nodes and one task per node,
you have been allocated 2 nodes only.
I do not know if this is related to this issue.

Note if you use the machinefile, a00551 has two slots (since it
appears twice in the machinefile) but a00553 has 20 slots (since it
appears once in the machinefile, the number of slots is automatically
detected)

Can you run
mpirun --mca plm_base_verbose 10 ...
So we can confirm tm is used.


Before invoking mpirun, you might want to cleanup the ompi directory in 
/tmp


Cheers,

Gilles

Oswin Krause <oswin.kra...@ruhr-uni-bochum.de> wrote:

Hi,


I am currently trying to set up OpenMPI in torque. OpenMPI is build 
with

tm support. Torque is correctly assigning nodes and I can run
mpi-programs on single nodes just fine. the problem starts when
processes are split between nodes.

For example, I create an interactive session with torque and start a
program by

qsub -I -n -l nodes=3:ppn=1
mpirun --tag-output -display-map hostname

which leads to
[a00551.science.domain:15932] [[65415,0],1] bind() failed on error
Address already in use (98)
[a00551.science.domain:15932] [[65415,0],1] ORTE_ERROR_LOG: Error in
file oob_usock_component.c at line 228
 Data for JOB [65415,1] offset 0

 ========================   JOB MAP   ========================

 Data for node: a00551  Num slots: 2    Max slots: 0    Num procs: 2
        Process OMPI jobid: [65415,1] App: 0 Process rank: 0 Bound: socket
0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt
0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]
        Process OMPI jobid: [65415,1] App: 0 Process rank: 1 Bound: socket
1[core 10[hwt 0-1]], socket 1[core 11[hwt 0-1]], socket 1[core 12[hwt
0-1]], socket 1[core 13[hwt 0-1]], socket 1[core 14[hwt 0-1]], socket
1[core 15[hwt 0-1]], socket 1[core 16[hwt 0-1]], socket 1[core 17[hwt
0-1]], socket 1[core 18[hwt 0-1]], socket 1[core 19[hwt
0-1]]:[../../../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB/BB/BB]

 Data for node: a00553.science.domain   Num slots: 1    Max slots: 0    Num
procs: 1
        Process OMPI jobid: [65415,1] App: 0 Process rank: 2 Bound: socket
0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt
0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]

 =============================================================
[1,0]<stdout>:a00551.science.domain
[1,2]<stdout>:a00551.science.domain
[1,1]<stdout>:a00551.science.domain


if I login on a00551 and start using the hostfile generated by the
PBS_NODEFILE, everything works:

(from within the interactive session)
echo $PBS_NODEFILE
/var/lib/torque/aux//278.a00552.science.domain
cat $PBS_NODEFILE
a00551.science.domain
a00553.science.domain
a00551.science.domain

(from within the separate login)
mpirun --hostfile /var/lib/torque/aux//278.a00552.science.domain -np 3
--tag-output -display-map hostname

 Data for JOB [65445,1] offset 0

 ========================   JOB MAP   ========================

 Data for node: a00551  Num slots: 2    Max slots: 0    Num procs: 2
        Process OMPI jobid: [65445,1] App: 0 Process rank: 0 Bound: socket
0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt
0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]
        Process OMPI jobid: [65445,1] App: 0 Process rank: 1 Bound: socket
1[core 10[hwt 0-1]], socket 1[core 11[hwt 0-1]], socket 1[core 12[hwt
0-1]], socket 1[core 13[hwt 0-1]], socket 1[core 14[hwt 0-1]], socket
1[core 15[hwt 0-1]], socket 1[core 16[hwt 0-1]], socket 1[core 17[hwt
0-1]], socket 1[core 18[hwt 0-1]], socket 1[core 19[hwt
0-1]]:[../../../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB/BB/BB]

 Data for node: a00553.science.domain   Num slots: 20   Max slots: 0    Num
procs: 1
        Process OMPI jobid: [65445,1] App: 0 Process rank: 2 Bound: socket
0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt
0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]

 =============================================================
[1,0]<stdout>:a00551.science.domain
[1,2]<stdout>:a00553.science.domain
[1,1]<stdout>:a00551.science.domain

I am kind of lost whats going on here. Anyone having an idea? I am
seriously considering this to be the problem of kerberos
authentification that we have to work with, but I fail to see how this
should affect the sockets.

Best,
Oswin
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users






















					Reply via email to