Hi again,
Tim Prins schrieb:
Hi,
On Monday 01 October 2007 03:56:16 pm Dino Rossegger wrote:
Hi again,
Yes the error output is the same:
root@sun:~# mpirun --hostfile hostfile main
[sun:23748] [0,0,0] ORTE_ERROR_LOG: Timeout in file
base/pls_base_orted_cmds.c at line 275
[sun:23748] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at
line 1164
[sun:23748] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90
[sun:23748] ERROR: A daemon on node saturn failed to start as expected.
[sun:23748] ERROR: There may be more information available from
[sun:23748] ERROR: the remote shell (see above).
[sun:23748] ERROR: The daemon exited unexpectedly with status 255.
[sun:23748] [0,0,0] ORTE_ERROR_LOG: Timeout in file
base/pls_base_orted_cmds.c at line 188
[sun:23748] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at
line 1196
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons for this job.
Returned value Timeout instead of ORTE_SUCCESS.
--------------------------------------------------------------------------
Can you try:
mpirun --debug-daemons --hostfile hostfile main
Did it but it doesn't give me any special output (as far as I can see that)
Heres the output:
root@sun:~# mpirun --debug-daemons --hostfile hostfile ./main
Daemon [0,0,1] checking in as pid 27168 on host sun
[sun:27168] [0,0,1] orted_recv_pls: received message from [0,0
,0]
[sun:27168] [0,0,1] orted_recv_pls: received kill_local_procs
[sun:27167] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_b
ase_orted_cmds.c at line 275
[sun:27167] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_mo
dule.c at line 1164
[sun:27167] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp
.c at line 90
[sun:27167] ERROR: A daemon on node saturn failed to start as
expected.
[sun:27167] ERROR: There may be more information available fro
m
[sun:27167] ERROR: the remote shell (see above).
[sun:27167] ERROR: The daemon exited unexpectedly with status
255.
[sun:27168] [0,0,1] orted_recv_pls: received message from [0,0
,0]
[sun:27168] [0,0,1] orted_recv_pls: received exit
[sun:27167] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_b
ase_orted_cmds.c at line 188
[sun:27167] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_mo
dule.c at line 1196
--------------------------------------------------------------
------------
mpirun was unable to cleanly terminate the daemons for this jo
b. Returned value Timeout instead of ORTE_SUCCESS.
--------------------------------------------------------------
------------
This may give more output about the error. Also, try
mpirun -mca pls rsh -mca pls_rsh_debug 1 --hostfile hostfile main
Heres the output, but I cant decipher it ^^
root@sun:~# mpirun -mca pls rsh -mca pls_rsh_debug 1 --hostfil
e hostfile main
[sun:27175] pls:rsh: local csh: 0, local sh: 1
[sun:27175] pls:rsh: assuming same remote shell as local shell
[sun:27175] pls:rsh: remote csh: 0, remote sh: 1
[sun:27175] pls:rsh: final template argv:
[sun:27175] pls:rsh: /usr/bin/ssh <template> orted --bootp
roxy 1 --name <template> --num_procs 3 --vpid_start 0 --nodena
me <template> --universe root@sun:default-universe-27175 --nsr
eplica "0.0.0;tcp://192.168.1.254:4733;tcp://172.16.0.202:4733
" --gprreplica "0.0.0;tcp://192.168.1.254:4733;tcp://172.16.0.
202:4733"
[sun:27175] pls:rsh: launching on node sun
[sun:27175] pls:rsh: sun is a LOCAL node
[sun:27175] pls:rsh: changing to directory /root
[sun:27175] pls:rsh: executing: (/usr/local/bin/orted) orted -
-bootproxy 1 --name 0.0.1 --num_procs 3 --vpid_start 0 --noden
ame sun --universe root@sun:default-universe-27175 --nsreplica
"0.0.0;tcp://192.168.1.254:4733;tcp://172.16.0.202:4733" --gp
rreplica "0.0.0;tcp://192.168.1.254:4733;tcp://172.16.0.202:47
33" --set-sid [SSH_AGENT_PID=24793 TERM=xterm SHELL=/bin/bash
SSH_CLIENT=10.2.56.124 21001 22 SSH_TTY=/dev/pts/0 USER=root L
D_LIBRARY_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/b
in:/sbin:/bin:/opt/c3-4/:/usr/lib:/usr/local/lib SSH_AUTH_SOCK
=/tmp/ssh-sxbbH24792/agent.24792 MAIL=/var/mail/root PATH=/usr
/local/bin:/usr/bin:/bin:/usr/games:/opt/c3-4/:/usr/local/lib
PWD=/root LANG=en_US.UTF-8 SHLVL=1 HOME=/root LOGNAME=root SSH
_CONNECTION=10.2.56.124 21001 172.16.0.202 22 _=/usr/local/bin
/mpirun OMPI_MCA_rds_hostfile_path=hostfile orte-job-globals O
MPI_MCA_pls_rsh_debug=1 OMPI_MCA_seed=0]
[sun:27175] pls:rsh: launching on node saturn
[sun:27175] pls:rsh: saturn is a REMOTE node
[sun:27175] pls:rsh: executing: (//usr/bin/ssh) /usr/bin/ssh s
aturn orted --bootproxy 1 --name
0.0.2 --num_procs 3 --vpid_st
art 0 --nodename saturn --universe root@sun:default-universe-2
7175 --nsreplica
"0.0.0;tcp://192.168.1.254:4733;tcp://172.16.
0.202:4733" --gprreplica
"0.0.0;tcp://192.168.1.254:4733;tcp:/
/172.16.0.202:4733" [SSH_AGENT_PID=24793 TERM=xterm
SHELL=/bin
/bash
SSH_CLIENT=10.2.56.124 21001 22 SSH_TTY=/dev/pts/0 USER=
root
LD_LIBRARY_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:
/usr/bin:/sbin:/bin:/opt/c3-4/:/usr/lib:/usr/local/lib SSH_AUT
H_SOCK=/tmp/ssh-sxbbH24792/agent.24792 MAIL=/var/mail/root PAT
H=/usr/local/bin:/usr/bin:/bin:/usr/games:/opt/c3-4/:/usr/loca
l/lib PWD=/root LANG=en_US.UTF-8
SHLVL=1 HOME=/root LOGNAME=ro
ot SSH_CONNECTION=10.2.56.124 21001 172.16.0.202 22 _=/usr/loc
al/bin/mpirun
OMPI_MCA_rds_hostfile_path=hostfile orte-job-glo
bals OMPI_MCA_pls_rsh_debug=1 OMPI_MCA_seed=0]
[sun:27175] [0,0,0] ORTE_ERROR_LOG: Timeout in file
base/pls_base_orted_cmds.c at line 275
[sun:27175] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at
line 1164
[sun:27175] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90
[sun:27175] ERROR: A daemon on node saturn failed to start as expected.
[sun:27175] ERROR: There may be more information available from
[sun:27175] ERROR: the remote shell (see above).
[sun:27175] ERROR: The daemon exited unexpectedly with status 255.
[sun:27175] [0,0,0] ORTE_ERROR_LOG: Timeout in file
base/pls_base_orted_cmds.c at line 188
[sun:27175] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at
line 1196
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons for this job.
Returned value Timeout instead of ORTE_SUCCESS.
--------------------------------------------------------------------------
This will print out the exact command that is used to launch the orted.
Also, I would highly recommend not running Open MPI as root. It is just a bad
idea.
Yes I know, I'm doing it just now for testing.
I wrote the following to my .ssh/environment (on all machines)
LD_LIBRARY_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bi
n:/opt/c3-4/:/usr/lib:/usr/local/lib;
PATH=$PATH:/usr/local/lib;
export LD_LIBRARY_PATH;
export PATH;
and added the statement you told me to the ssd_config (on all machines):
PermitUserEnvironment yes
And it seems to me that the pathes are correct now.
My shell is bash (/bin/bash)
When running locate orted (to find out where exactly my openmpi
installation is (compilation defaults) i saw that, on sun there was a
/usr/bin/orted while there wasn't one on saturn.
I deleted /usr/bin/orted on sun and tried again with the option --prefix
/usr/local/ (which seems to be my installation directory) but it
didn't work (same error).
Is it possible that you are mixing 2 different installations of Open MPI? You
may consider installing OpenMPI to a NFS drive to make these things a bit
easier.
Is there a script or anything like that with which I can uninstall
openmpi, because i'll might try a new compilation to /opt/openmpi since
it doesn't look like I would be able to solve the problem.
If you still have the tree around that you used to 'make' Open MPI, you can
just go into that tree and type 'make uninstall'.
Hope this helps,
Tim
jody schrieb:
Now that the PATHs seem to be set correctly for
ssh i don't know what the problem could be.
Is the error message still the same on as in the first mail?
Did you do the envorpnment/sshd_config on both machines?
What shell are you using?
On other test you could make is to start your application
with the --prefix option:
$mpirun -np 2 --prefix /opt/openmpi -H sun,saturn ./main
(assuming your Open MPI installation lies in /opt/openmpi
on both machines)
Jody
On 10/1/07, Dino Rossegger <dino.rosseg...@gmx.at> wrote:
Hi Jodi,
did the steps as you said, but it didn't work for me.
I set LD_LIBRARY_PATH in /etc/environment and ~/.shh/environment and
made the changes to sshd_config.
But this all didn't solve my problem, although the pahts seemed to be
set correctly (judging what ssh saturn `printenv >> test` says). I also
restarted the ssh server, the error is the same.
Hope you can help me out here and thanks for your help so far
dino
jody schrieb:
Dino -
I had a similar problem.
I was only able to solve it by setting PATH and LS_LIBRARY_PATH
in the file ~/ssh/environment on the client and setting
PermitUserEnvironment yes
in /etc/ssh/sshd_config on the server (for this you need root
prioviledge though)
To be on the safe side, i did both on all my nodes
Jody
On 9/27/07, Dino Rossegger <dino.rosseg...@gmx.at> wrote:
Hi Jody,
Thanks for your help, it really is the case that either in PATH nor in
LD_LIBRARY_PATH the path to the libs is set correctly. I'll try out,
hope it works.
jody schrieb:
Hi Dino
Try
ssh saturn printenv | grep PATH
>from your host sun to see what your environment variables are when
ssh is run without a shell.
On 9/27/07, Dino Rossegger <dino.rosseg...@gmx.at> wrote:
Hi,
I have a problem running a simple programm mpihello.cpp.
Here is a excerp of the error and the command
root@sun:~# mpirun -H sun,saturn main
[sun:25213] [0,0,0] ORTE_ERROR_LOG: Timeout in file
base/pls_base_orted_cmds.c at line 275
[sun:25213] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c
at line 1164
[sun:25213] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at
line 90 [sun:25213] ERROR: A daemon on node saturn failed to start
as expected. [sun:25213] ERROR: There may be more information
available from [sun:25213] ERROR: the remote shell (see above).
[sun:25213] ERROR: The daemon exited unexpectedly with status 255.
[sun:25213] [0,0,0] ORTE_ERROR_LOG: Timeout in file
base/pls_base_orted_cmds.c at line 188
[sun:25213] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c
at line 1196
--------------------------------------------------------------------
------ mpirun was unable to cleanly terminate the daemons for this
job. Returned value Timeout instead of ORTE_SUCCESS.
--------------------------------------------------------------------
------
The program is runable from each node alone (mpirun -np2 main)
My PathVariables:
$PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/c3
-4/:/usr/lib:/usr/local/libecho $LD_LIBRARY_PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/c3
-4/:/usr/lib:/usr/local/lib
Passwordless ssh is up 'n running
I walked through the FAQ and Mailing Lists but couldn't find any
solution for my problem.
Thanks
Dino R.
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
