Re: [OMPI users] Performance issue of mpirun/mpi_init

[Ralph Castain]

I shaved about 30% off the time - the patch is waiting for 1.8.1, but you can 
try it now (see the ticket for the changeset):

https://svn.open-mpi.org/trac/ompi/ticket/4510#comment:1

I've added you to the ticket so you can follow what I'm doing. Getting any 
further improvement will take a little longer due to travel and vacation, but 
I'll keep poking at it.

Ralph

On Apr 10, 2014, at 10:25 AM, Victor Vysotskiy  
wrote:

> Hi again,
> 
> > Okay, I'll try to do a little poking around. Meantime, please send along 
> > the output from >"ompi_info" so we can see how this was configured and what 
> > built.
> 
> enclosed please find the requested information. It would be great to have an 
> workaround for 1.8 because with 1.8 our  verification suite takes (6.2 hrs) 
> 2x times longer to complete compared to 1.6.5  (3 hrs).
> 
> With best regards,
> Victor.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Optimal mapping/binding when threads are used?

[Ralph Castain]

Interesting data. Couple of quick points that might help:

option B is equivalent to --map-by node --bind-to none. When you bind to every 
core on the node, we don't bind you at all since "bind to all" is exactly 
equivalent to "bind to none". So it will definitely run slower as the threads 
run across the different NUMA regions on the node.

You might also want to try --map-by socket, with no binding directive. This 
would map one process to each socket, binding it to the socket - which is 
similar to what your option A actually accomplished. The only difference is 
that the procs that share a node will differ in rank by 1, whereas option A 
would have those procs differ in rank by N. Depending on your communication 
pattern, this could make a big difference.

Map-by socket is typically the fastest performance for threaded apps. You 
generally don't want P=1 unless you have a *lot* of threads in the process as 
it removes any use of shared memory, and so messaging will run slower - and you 
want the ranks that share a node to be the ones that most frequently 
communicate to each other, if you can identify them.

HTH
Ralph

On Apr 10, 2014, at 5:59 PM, Saliya Ekanayake  wrote:

> Hi,
> 
> I am evaluating the performance of a clustering program written in Java with 
> MPI+threads and would like to get some insight in solving a peculiar case. 
> I've attached a performance graph to explain this.
> 
> In essence the tests were carried out as TxPxN, where T is threads per 
> process, P is processes per node, and N is number of nodes. I noticed an 
> inefficiency with Tx1xN cases in general (tall bars in graph).
> 
> To elaborate a bit further, 
> 1. each node has 2 sockets with 4 cores each (totaling 8 cores) 
> 2. used OpenMPI 1.7.5rc5 (later tested with 1.8 and observed the same)
> 3. with options
>  A.) --map-by node:PE=4 and --bind-to core
>  B.) --map-by node:PE=8 and --bind-to-core
>  C.) --map-by socket and --bind-to none
> 
> Timing of A,B,C came out as A < B < C, so used results from option A for 
> Tx1xN in the graph. 
> 
> Could you please give some suggestion that may help to speed up these Tx1xN 
> cases? Also, I expected B to perform better than A as threads could utilize 
> all 8 cores, but it wasn't the case.
> 
> Thank you,
> Saliya
> 
> 
> 
> 
> -- 
> Saliya Ekanayake esal...@gmail.com 
> Cell 812-391-4914 Home 812-961-6383
> http://saliya.org
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Troubleshooting mpirun with tree spawn hang

[Anthony Alba]

Is there a way to troubleshoot
plm_rsh_no_tree_spawn=true hang?

I have a set of passwordless-ssh nodes, each node can ssh into any other.,
i.e.,

for h1 in A B C D; do for h2 in A B C D; do ssh $h1 ssh $h2 hostname; done;
done

works perfectly.

Generally tree spawn works, however there is one host where
launching  mpirun with tree spawn hangs as soon as there are 6 or more host
(with launch node also in the host list). If the launcher is not in the
host list the hang happens with five hosts.


- Anthony

Re: [OMPI users] Optimal mapping/binding when threads are used?

[Saliya Ekanayake]

Thank you Ralph for the details and it's a good point you mentioned on
mapping by node vs socket. We have another program that uses a chain of
send receives, which will benefit from having consecutive ranks nearby.

I've a question on bind to none being equal to bind to all. I understand
the two concepts mean the same thing, but I remember seeing poor
performance when bind to none is explicitly given. I need to check the
options I used and will let you know.

Yes, this test was mainly to understand how different patterns perform and
it seems P=1 is not suitable for this hardware configuration and may be in
general as you've mentioned.

Thank you,
Saliya


On Fri, Apr 11, 2014 at 12:30 AM, Ralph Castain  wrote:

> Interesting data. Couple of quick points that might help:
>
> option B is equivalent to --map-by node --bind-to none. When you bind to
> every core on the node, we don't bind you at all since "bind to all" is
> exactly equivalent to "bind to none". So it will definitely run slower as
> the threads run across the different NUMA regions on the node.
>
> You might also want to try --map-by socket, with no binding directive.
> This would map one process to each socket, binding it to the socket - which
> is similar to what your option A actually accomplished. The only difference
> is that the procs that share a node will differ in rank by 1, whereas
> option A would have those procs differ in rank by N. Depending on your
> communication pattern, this could make a big difference.
>
> Map-by socket is typically the fastest performance for threaded apps. You
> generally don't want P=1 unless you have a *lot* of threads in the process
> as it removes any use of shared memory, and so messaging will run slower -
> and you want the ranks that share a node to be the ones that most
> frequently communicate to each other, if you can identify them.
>
> HTH
> Ralph
>
> On Apr 10, 2014, at 5:59 PM, Saliya Ekanayake  wrote:
>
> Hi,
>
> I am evaluating the performance of a clustering program written in Java
> with MPI+threads and would like to get some insight in solving a peculiar
> case. I've attached a performance graph to explain this.
>
> In essence the tests were carried out as TxPxN, where T is threads per
> process, P is processes per node, and N is number of nodes. I noticed an
> inefficiency with Tx*1*xN cases in general (tall bars in graph).
>
> To elaborate a bit further,
> 1. each node has 2 sockets with 4 cores each (totaling 8 cores)
> 2. used OpenMPI 1.7.5rc5 (later tested with 1.8 and observed the same)
> 3. with options
>  A.) --map-by node:PE=4 and --bind-to core
>  B.) --map-by node:PE=8 and --bind-to-core
>  C.) --map-by socket and --bind-to none
>
> Timing of A,B,C came out as A < B < C, so used results from option A for Tx
> *1*xN in the graph.
>
> Could you please give some suggestion that may help to speed up these Tx
> *1*xN cases? Also, I expected B to perform better than A as threads could
> utilize all 8 cores, but it wasn't the case.
>
> Thank you,
> Saliya
>
>
> 
>
> --
> Saliya Ekanayake esal...@gmail.com
> Cell 812-391-4914 Home 812-961-6383
> http://saliya.org
>  ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Saliya Ekanayake esal...@gmail.com
Cell 812-391-4914 Home 812-961-6383
http://saliya.org

[OMPI users] can't run mpi-jobs on remote host

[Lubrano Francesco]

Dear MPI users,

I have a problem with open-mpi (version 1.8).

I'm just beginning to undestand how mpi works and I can't find solution of my 
problem on FAQ page.

I have two machines (a local host and a remote host) with linux open-suse 
(latest version) and open-mpi 1.8

I can run open-mpi jobs on both machines from theirself (in a "local" way).

I have not connections problem: ssh from the first to the second works 
correctly and I can run programs on remote machine.

If I run a simple non mpi program on the remot host from the local one, it 
doesnt' work. I think it is a buffer pointer problem (status 1). I didn't 
change open-mpi settings and PATH is correct. I have just one open-mpi versione 
on both of them. Open-mpi doesn't show any error: just return to the local 
machine.

I can also request to run a false program: it does not change. on the terminal 
I can read


francesco@linux-hldu:~> mpirun -host Frank@158.110.39.110 hostname

Password:

francesco@linux-hldu:~>


If I request debug, the result is:


francesco@linux-hldu:~> mpirun -d --host Frank@158.110.39.110 hostname
[linux-hldu.site:02537] sess_dir_finalize: job session dir not empty - leaving
[linux-hldu.site:02537] procdir: 
/tmp/openmpi-sessions-francesco@linux-hldu_0/33429/0/0
[linux-hldu.site:02537] jobdir: 
/tmp/openmpi-sessions-francesco@linux-hldu_0/33429/0
[linux-hldu.site:02537] top: openmpi-sessions-francesco@linux-hldu_0
[linux-hldu.site:02537] tmp: /tmp
Password:
[linux-o5sl.site:04273] sess_dir_finalize: job session dir not empty - leaving
[linux-o5sl.site:04273] procdir: 
/tmp/openmpi-sessions-Frank@linux-o5sl_0/33429/0/1
[linux-o5sl.site:04273] jobdir: /tmp/openmpi-sessions-Frank@linux-o5sl_0/33429/0
[linux-o5sl.site:04273] top: openmpi-sessions-Frank@linux-o5sl_0
[linux-o5sl.site:04273] tmp: /tmp
[linux-o5sl.site:04273] sess_dir_finalize: job session dir not empty - leaving
exiting with status 1
[linux-hldu.site:02537] sess_dir_finalize: job session dir not empty - leaving
exiting with status 1


Do you understand where is the problem? How can I get more information?

Thank you for your cooperation


regards


Francesco

Re: [OMPI users] OpenMPI 1.8.0 + PGI 13.6 = undeclared variable __LDBL_MANT_DIG__

[Jeff Squyres (jsquyres)]

On Apr 9, 2014, at 8:47 PM, Filippo Spiga  wrote:

> I haven't solve this yet but I managed to move to code to be compatible woth 
> PGI 14.3. Open MPI 1.8 compiles perfectly with the latest PGI.
> 
> In parallel I will push this issue to the PGI forum. 

FWIW: We've seen this kind of issue before with older versions of compilers, 
i.e., compilers sometimes have bugs, too.  :-\

Glad you've got it working with the new version of PGI.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] File locking in ADIO, OpenMPI 1.6.4

[Jeff Squyres (jsquyres)]

Sorry for the delay in replying.

Can you try upgrading to Open MPI 1.8, which was released last week?  We 
refreshed the version of ROMIO that is included in OMPI 1.8 vs. 1.6.


On Apr 8, 2014, at 6:49 PM, Daniel Milroy  wrote:

> Hello,
>  
> Recently a couple of our users have experienced difficulties with compute 
> jobs failing with OpenMPI 1.6.4 compiled against GCC 4.7.2, with the nodes 
> running kernel 2.6.32-279.5.2.el6.x86_64.  The error is:
>  
> File locking failed in ADIOI_Set_lock(fd 7,cmd F_SETLKW/7,type 
> F_WRLCK/1,whence 0) with return value  and errno 26.
> - If the file system is NFS, you need to use NFS version 3, ensure that the 
> lockd daemon is running on all the machines, and mount the directory with the 
> 'noac' option (no attribute caching).
> - If the file system is LUSTRE, ensure that the directory is mounted with the 
> 'flock' option.
> ADIOI_Set_lock:: Function not implemented
> ADIOI_Set_lock:offset 0, length 8
>  
> The file system in question is indeed Lustre, and mounting with flock isn’t 
> possible in our environment.  I recommended the following changes to the 
> users’ code:
>  
> MPI_Info_set(info, "collective_buffering", "true");
> MPI_Info_set(info, "romio_lustre_ds_in_coll", "disable");
> MPI_Info_set(info, "romio_ds_read", "disable");
> MPI_Info_set(info, "romio_ds_write", "disable");
>  
> Which results in the same error as before.  Are there any other MPI options I 
> can set?
>  
>  
> Thank you in advance for any advice,
>  
> Dan Milroy
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Troubleshooting mpirun with tree spawn hang

[Ralph Castain]

I'm a little confused - the "no_tree_spawn=true" option means that we are *not* 
using tree spawn, and so mpirun is directly launching each daemon onto its 
node. Thus, this requires that the host mpirun is on be able to ssh to every 
other host in the allocation.

You can debug the rsh launcher by setting "-mca plm_base_verbose 5 
--debug-daemons" on the cmd line.


On Apr 10, 2014, at 9:50 PM, Anthony Alba  wrote:

> 
> Is there a way to troubleshoot 
> plm_rsh_no_tree_spawn=true hang?
> 
> I have a set of passwordless-ssh nodes, each node can ssh into any other., 
> i.e., 
> 
> for h1 in A B C D; do for h2 in A B C D; do ssh $h1 ssh $h2 hostname; done; 
> done 
> 
> works perfectly.
> 
> Generally tree spawn works, however there is one host where
> launching  mpirun with tree spawn hangs as soon as there are 6 or more host 
> (with launch node also in the host list). If the launcher is not in the host 
> list the hang happens with five hosts.
> 
> 
> - Anthony
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] can't run mpi-jobs on remote host

[Ralph Castain]

Please see:

http://www.open-mpi.org/faq/?category=rsh#ssh-keys

short answer: you need to be able to ssh to the remote hosts without a password


On Apr 11, 2014, at 1:09 AM, Lubrano Francesco 
 wrote:

> Dear MPI users,
> I have a problem with open-mpi (version 1.8).
> I'm just beginning to undestand how mpi works and I can't find solution of my 
> problem on FAQ page.
> I have two machines (a local host and a remote host) with linux open-suse 
> (latest version) and open-mpi 1.8
> I can run open-mpi jobs on both machines from theirself (in a "local" way).
> I have not connections problem: ssh from the first to the second works 
> correctly and I can run programs on remote machine.
> If I run a simple non mpi program on the remot host from the local one, it 
> doesnt' work. I think it is a buffer pointer problem (status 1). I didn't 
> change open-mpi settings and PATH is correct. I have just one open-mpi 
> versione on both of them. Open-mpi doesn't show any error: just return to the 
> local machine.
> I can also request to run a false program: it does not change. on the 
> terminal I can read
> 
> francesco@linux-hldu:~> mpirun -host Frank@158.110.39.110 hostname
> Password: 
> francesco@linux-hldu:~>
> 
> If I request debug, the result is:
> 
> francesco@linux-hldu:~> mpirun -d --host Frank@158.110.39.110 hostname
> [linux-hldu.site:02537] sess_dir_finalize: job session dir not empty - leaving
> [linux-hldu.site:02537] procdir: 
> /tmp/openmpi-sessions-francesco@linux-hldu_0/33429/0/0
> [linux-hldu.site:02537] jobdir: 
> /tmp/openmpi-sessions-francesco@linux-hldu_0/33429/0
> [linux-hldu.site:02537] top: openmpi-sessions-francesco@linux-hldu_0
> [linux-hldu.site:02537] tmp: /tmp
> Password: 
> [linux-o5sl.site:04273] sess_dir_finalize: job session dir not empty - leaving
> [linux-o5sl.site:04273] procdir: 
> /tmp/openmpi-sessions-Frank@linux-o5sl_0/33429/0/1
> [linux-o5sl.site:04273] jobdir: 
> /tmp/openmpi-sessions-Frank@linux-o5sl_0/33429/0
> [linux-o5sl.site:04273] top: openmpi-sessions-Frank@linux-o5sl_0
> [linux-o5sl.site:04273] tmp: /tmp
> [linux-o5sl.site:04273] sess_dir_finalize: job session dir not empty - leaving
> exiting with status 1
> [linux-hldu.site:02537] sess_dir_finalize: job session dir not empty - leaving
> exiting with status 1
> 
> Do you understand where is the problem? How can I get more information?
> Thank you for your cooperation
> 
> regards
> 
> Francesco 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] mpirun problem when running on more than three hosts with OpenMPI 1.8

[Allan Wu]

Hello everyone,

I am running a simple helloworld program on several nodes using OpenMPI
1.8. Running commands on single node or small number of nodes are
successful, but when I tried to run the same binary on four different
nodes, problems occurred.

I am using 'mpirun' command line like the following:
# mpirun --prefix /mnt/embedded_root/openmpi -np 4 --map-by node -hostfile
hostfile ./helloworld
And my hostfile looks something like these:
10.0.0.16
10.0.0.17
10.0.0.18
10.0.0.19

When executing this command, it will result in an error message "sh: syntax
error: unexpected word", and the program will deadlock. When I added
"--debug-devel" the output is in the attachment "err_msg_0.txt". In the
log, "fpga0" is the hostname of "10.0.0.16" and "fpga1" is for "10.0.0.17"
and so on.

However, the weird part is that after I remove one line in the hostfile,
the problem goes away. It does not matter which host I remove, as long as
there is less than four hosts, the program can execute without any problem.

I also tried using hostname in the hostfile, as:
fpga0
fpga1
fpga2
fpga3
And the same problem occurs, and the error message becomes "Host key
verification failed.". I have setup public/private key pairs on all nodes,
and each node can ssh to any node without problems. I also attached the
message of --debug-devel as "err_msg_1.txt".

I'm running MPI programs on embedded ARM processors. I have previously
posted questions on cross-compilation on the develop mailing list, which
contains the setup I used. If you need the information please refer to
http://www.open-mpi.org/community/lists/devel/2014/04/14440.php, and the
output of 'ompi-info --all' is also attached with this email.

Please let me know if I need to provide more information. Thanks in advance!

Regards,
--
Di Wu (Allan)
PhD student, VAST Laboratory ,
Department of Computer Science, UC Los Angeles
Email: al...@cs.ucla.edu
[fpga0:02057] sess_dir_finalize: job session dir not empty - leaving
[fpga0:02057] procdir: /tmp/openmpi-sessions-root@fpga0_0/16688/0/0
[fpga0:02057] jobdir: /tmp/openmpi-sessions-root@fpga0_0/16688/0
[fpga0:02057] top: openmpi-sessions-root@fpga0_0
[fpga0:02057] tmp: /tmp
[fpga0:02057] mpirun: reset PATH: 
/mnt/embedded_root/openmpi/bin:/mnt/embedded_root/openmpi/bin:/sbin:/usr/sbin:/bin:/usr/bin
[fpga0:02057] mpirun: reset LD_LIBRARY_PATH: 
/mnt/embedded_root/openmpi/lib:/mnt/embedded_root/openmpi/lib:/mnt/embedded_root/lib
[fpga2:01321] sess_dir_finalize:i job session dir not empty - leaving
[fpga1:01882] sess_dir_finalize: job session dir not empty - leaving
[fpga2:01321] procdir: /tmp/openmpi-sessions-root@fpga2_0/16688/0/2
[fpga2:01321] jobdir: /tmp/openmpi-sessions-root@fpga2_0/16688/0
[fpga2:01321] top: openmpi-sessions-root@fpga2_0
[fpga2:01321] tmp: /tmp
[fpga1:01882] procdir: /tmp/openmpi-sessions-root@fpga1_0/16688/0/1
[fpga1:01882] jobdir: /tmp/openmpi-sessions-root@fpga1_0/16688/0
[fpga1:01882] top: openmpi-sessions-root@fpga1_0
[fpga1:01882] tmp: /tmp
sh: syntax error: unexpected word
[fpga2:01321] sess_dir_finalize: job session dir not empty - leaving
exiting with status 0
[fpga1:01882] sess_dir_finalize: job session dir not empty - leaving
exiting with status 1
^Cmpirun: abort is already in progress...hit ctrl-c again to forcibly terminate

[fpga0:02057] sess_dir_finalize: job session dir not empty - leaving

[fpga0:02104] sess_dir_finalize: job session dir not empty - leaving
[fpga0:02104] procdir: /tmp/openmpi-sessions-root@fpga0_0/16641/0/0
[fpga0:02104] jobdir: /tmp/openmpi-sessions-root@fpga0_0/16641/0
[fpga0:02104] top: openmpi-sessions-root@fpga0_0
[fpga0:02104] tmp: /tmp
[fpga0:02104] mpirun: reset PATH: 
/mnt/embedded_root/openmpi/bin:/mnt/embedded_root/openmpi/bin:/sbin:/usr/sbin:/bin:/usr/bin
[fpga0:02104] mpirun: reset LD_LIBRARY_PATH: 
/mnt/embedded_root/openmpi/lib:/mnt/embedded_root/openmpi/lib:/mnt/embedded_root/lib
[fpga2:01361] sess_dir_finalize: job session dir not empty - leaving
[fpga1:01926] sess_dir_finalize: job session dir not empty - leaving
[fpga1:01926] procdir: /tmp/openmpi-sessions-root@fpga1_0/16641/0/1
[fpga1:01926] jobdir: /tmp/openmpi-sessions-root@fpga1_0/16641/0
[fpga1:01926] top: openmpi-sessions-root@fpga1_0
[fpga1:01926] tmp: /tmp
[fpga2:01361] procdir: /tmp/openmpi-sessions-root@fpga2_0/16641/0/2
[fpga2:01361] jobdir: /tmp/openmpi-sessions-root@fpga2_0/16641/0
[fpga2:01361] top: openmpi-sessions-root@fpga2_0
[fpga2:01361] tmp: /tmp
Host key verification failed.
[fpga2:01361] sess_dir_finalize: job session dir not empty - leaving
[fpga1:01926] sess_dir_finalize: job session dir not empty - leaving
exiting with status 0
exiting with status 1
^Cmpirun: abort is already in progress...hit ctrl-c again to forcibly terminate

[fpga0:02104] sess_dir_finalize: job session dir not empty - leaving


log.tar.gz
Description: GNU Zip compressed data

Re: [OMPI users] mpirun problem when running on more than three hosts with OpenMPI 1.8

[Ralph Castain]

The problem is with the tree-spawn nature of the rsh/ssh launcher. For 
scalability, mpirun only launches a first "layer" of daemons. Each of those 
daemons then launches another layer in a tree-like fanout. The default pattern 
is such that you first notice it when you have four nodes in your allocation.

You have two choices:

* you can just add the MCA param plm_rsh_no_tree_spawn=1 to your 
environment/cmd line

* you can resolve the tree spawn issue so that a daemon on one of your nodes is 
capable of ssh-ing a daemon on another node

Either way will work.
Ralph


On Apr 11, 2014, at 11:17 AM, Allan Wu  wrote:

> Hello everyone,
> 
> I am running a simple helloworld program on several nodes using OpenMPI 1.8. 
> Running commands on single node or small number of nodes are successful, but 
> when I tried to run the same binary on four different nodes, problems 
> occurred. 
> 
> I am using 'mpirun' command line like the following:
> # mpirun --prefix /mnt/embedded_root/openmpi -np 4 --map-by node -hostfile 
> hostfile ./helloworld
> And my hostfile looks something like these:
> 10.0.0.16
> 10.0.0.17
> 10.0.0.18
> 10.0.0.19
> 
> When executing this command, it will result in an error message "sh: syntax 
> error: unexpected word", and the program will deadlock. When I added 
> "--debug-devel" the output is in the attachment "err_msg_0.txt". In the log, 
> "fpga0" is the hostname of "10.0.0.16" and "fpga1" is for "10.0.0.17" and so 
> on.
> 
> However, the weird part is that after I remove one line in the hostfile, the 
> problem goes away. It does not matter which host I remove, as long as there 
> is less than four hosts, the program can execute without any problem. 
> 
> I also tried using hostname in the hostfile, as:
> fpga0
> fpga1
> fpga2
> fpga3
> And the same problem occurs, and the error message becomes "Host key 
> verification failed.". I have setup public/private key pairs on all nodes, 
> and each node can ssh to any node without problems. I also attached the 
> message of --debug-devel as "err_msg_1.txt". 
> 
> I'm running MPI programs on embedded ARM processors. I have previously posted 
> questions on cross-compilation on the develop mailing list, which contains 
> the setup I used. If you need the information please refer to 
> http://www.open-mpi.org/community/lists/devel/2014/04/14440.php, and the 
> output of 'ompi-info --all' is also attached with this email.
> 
> Please let me know if I need to provide more information. Thanks in advance!
> 
> Regards,
> --
> Di Wu (Allan)
> PhD student, VAST Laboratory,
> Department of Computer Science, UC Los Angeles
> Email: al...@cs.ucla.edu
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Troubleshooting mpirun with tree spawn hang

[Anthony Alba]

Ooops I meant = false.

Thanks for the tip, it turns out the fault lay in a specific node that
required oob_tcp_if_include to be set.

On Friday, 11 April 2014, Ralph Castain  wrote:

> I'm a little confused - the "no_tree_spawn=true" option means that we are
> *not* using tree spawn, and so mpirun is directly launching each daemon
> onto its node. Thus, this requires that the host mpirun is on be able to
> ssh to every other host in the allocation.
>
> You can debug the rsh launcher by setting "-mca plm_base_verbose 5
> --debug-daemons" on the cmd line.
>
>
> On Apr 10, 2014, at 9:50 PM, Anthony Alba 
> >
> wrote:
>
> >
> > Is there a way to troubleshoot
> > plm_rsh_no_tree_spawn=true hang?
> >
> > I have a set of passwordless-ssh nodes, each node can ssh into any
> other., i.e.,
> >
> > for h1 in A B C D; do for h2 in A B C D; do ssh $h1 ssh $h2 hostname;
> done; done
> >
> > works perfectly.
> >
> > Generally tree spawn works, however there is one host where
> > launching  mpirun with tree spawn hangs as soon as there are 6 or more
> host (with launch node also in the host list). If the launcher is not in
> the host list the hang happens with five hosts.
> >
> >
> > - Anthony
> >
> > ___
> > users mailing list
> > us...@open-mpi.org 
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ___
> users mailing list
> us...@open-mpi.org 
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

[OMPI users] OpenMPI PMI2 with SLURM 14.03 not working

[Anthony Alba]

Not sure if this is a SLURM or OMPI issue so please bear with the
cross-posting...

The OpenMPI FAQ mentions an issue with slurm 2.6.3/pmi2.
https://www.open-mpi.org/faq/?category=slurm#slurm-2.6.3-issue

I have built both 1.7.5/1.8 against slurm 14.03/pmi2.

When I launch openmpi/examples/hello_c on a single node allocation:

srun --mpi=pmi2 -N 1 hello_c:

srun -N 1 --mpi=pmi2 hello_c
srun: error: _server_read: fd 18 got error or unexpected eof reading header
srun: error: step_launch_notify_io_failure: aborting, io error with
slurmstepd on node 0
srun: Job step aborted: Waiting up to 2 seconds for job step to finish.
srun: error: Timed out waiting for job step to complete


with --slurmd-debug=9: (I'm not sure what is the meaning of "ip
111.110.61.48 sd 14"
below, is that ip as in ip address? It is not the ip address of any Nodes
in my partition)

slurmstepd: mpi/pmi2: client_resp_send: 26cmd=kvs-put-response;rc=0;
slurmstepd: mpi/pmi2: _tree_listen_readable
slurmstepd: mpi/pmi2: _task_readable
slurmstepd: mpi/pmi2: got client request: 14 cmd=kvs-fence;
slurmstepd: mpi/pmi2: _tree_listen_readable
slurmstepd: mpi/pmi2: _task_readable
slurmstepd: mpi/pmi2: _tree_listen_read
slurmstepd: _tree_listen_read: accepted tree connection: ip 111.110.61.48
sd 14
slurmstepd: _handle_accept_rank: going to read() client rank
slurmstepd: _handle_accept_rank: got client rank 1478164480 on fd 14
srun: error: _server_read: fd 18 got error or unexpected eof reading header
srun: error: step_launch_notify_io_failure: aborting, io error with
slurmstepd on node 0
srun: Job step aborted: Waiting up to 2 seconds for job step to finish.
srun: error: Timed out waiting for job step to complete

Launching with salloc/sbatch works.

- Anthony

[OMPI users] Question on suspending/resuming MPI processes with SIGSTOP

[Frank Wein]

Hi,
I've got a question on suspending/resuming an process started with
"mpirun", I've already found the FAQ entry on this
http://www.open-mpi.de/faq/?category=running#suspend-resume but I've
still got a question on this. Basically for now let's assume I'm running
all MPI processes on one host only with one multi-core CPU (so I could
directly send SIGSTOP to other processes if I want to). What I wonder
about is the following: I want to start multiple (let's say four)
instances of my program with "mpirun -np 4 ./mybinary" and at some point
during the program execution I want to suspend two of those four
processes, those two processes are waiting at an MPI_Barrier() at this
point. The goal of that is to suspend execution so that those processes
don't use the CPU at all while they are suspended (that's not the case
with MPI_Barrier as far as I understand this). So now my question
basically is: Will it work when I send SIGSTOP signal from my MPI rank 0
process to these two processes while they are waiting at an MPI_Barrier
and then those two processes won't use the CPU anymore? Later I want to
resume the processes with SIGCONT when the other two processes also
arrived at this MPI_Barrier. Performance of the barrier does not matter
here, what matters for me is that those suspended processes don't cause
any CPU usage. I never used SIGSTOP signal so far, so I'm not sure if
this will work. And before I start coding the logic for this into my
program, I thought I'll ask here first if this will work at all :).

Frank

Re: [OMPI users] Question on suspending/resuming MPI processes with SIGSTOP

[Frank Wein]

Frank Wein wrote:
[...]

Or basically my question could also be rephrased as: Is there a barrier
mechanism I could use in OMPI that causes basically very few to no CPU
usage (with higher latency then)? Intel MPI for example seems to have
the env var "I_MPI_WAIT_MODE=1" which uses some wait mechanism instead
of polling for a barrier. My idea for OMPI was to use SIGSTOP/SIGCONT in
combination with MPI_Barrier, but I'm open for other suggestions as well
if those work better (or at all).

Frank

Re: [OMPI users] Question on suspending/resuming MPI processes with SIGSTOP

[Ralph Castain]

I'm afraid our suspend/resume support only allows the signal to be applied to 
*all* procs, not selectively to some. For that matter, I'm unaware of any 
MPI-level API for hitting a proc with a signal - so I'm not sure how you would 
programmatically have rank0 suspend some other ranks.

On Apr 11, 2014, at 12:16 PM, Frank Wein  wrote:

> Hi,
> I've got a question on suspending/resuming an process started with
> "mpirun", I've already found the FAQ entry on this
> http://www.open-mpi.de/faq/?category=running#suspend-resume but I've
> still got a question on this. Basically for now let's assume I'm running
> all MPI processes on one host only with one multi-core CPU (so I could
> directly send SIGSTOP to other processes if I want to). What I wonder
> about is the following: I want to start multiple (let's say four)
> instances of my program with "mpirun -np 4 ./mybinary" and at some point
> during the program execution I want to suspend two of those four
> processes, those two processes are waiting at an MPI_Barrier() at this
> point. The goal of that is to suspend execution so that those processes
> don't use the CPU at all while they are suspended (that's not the case
> with MPI_Barrier as far as I understand this). So now my question
> basically is: Will it work when I send SIGSTOP signal from my MPI rank 0
> process to these two processes while they are waiting at an MPI_Barrier
> and then those two processes won't use the CPU anymore? Later I want to
> resume the processes with SIGCONT when the other two processes also
> arrived at this MPI_Barrier. Performance of the barrier does not matter
> here, what matters for me is that those suspended processes don't cause
> any CPU usage. I never used SIGSTOP signal so far, so I'm not sure if
> this will work. And before I start coding the logic for this into my
> program, I thought I'll ask here first if this will work at all :).
> 
> Frank
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] OpenMPI PMI2 with SLURM 14.03 not working [SOLVED]

[Anthony Alba]

Answered in the slurm-devel list: it is a bug in SLURM 14.03.

The fix is already in HEAD and also will be in 14.03.1

https://groups.google.com/forum/#!topic/slurm-devel/1ctPkEn7TFI

- Anthony