Re: [OMPI users] Forcing OpenMPI to use Ethernet interconnect instead of InfiniBand

[Muhammad Ansar Javed]

Thanks Jeff,
It worked. Now latency and bandwidth benchmarks are in performing as
expected for both Ethernet and InfiniBand.

--Ansar

On Wed, Sep 10, 2014 at 3:34 PM, Jeff Squyres (jsquyres)  wrote:

> Are you inadvertently using the MXM MTL?  That's an alternate Mellanox
> transport that may activate itself, even if you've disabled the openib
> BTL.  Try this:
>
>   mpirun --mca pml ob1 --mca btl ^openib ...
>
> This forces the use of the ob1 PML (which forces the use of the BTLs, not
> the MTLs), and then disables the openib BTL.
>
>
> On Sep 9, 2014, at 10:24 AM, Muhammad Ansar Javed <
> muhammad.an...@seecs.edu.pk> wrote:
>
> > Hi,
> >
> > I am currently conducting some testing on system with Gigabit and
> InfiniBand interconnects. Both Latency and Bandwidth benchmarks are doing
> well as expected on InfiniBand interconnects but Ethernet interconnect is
> achieving very high performance from expectations. Ethernet and InfiniBand
> both are achieving equivalent performance.
> >
> > For some reason, it looks like openmpi (v1.8.1) is using the InfiniBand
> interconnect rather than the Gigabit or TCP communication is being emulated
> to use InifiniBand interconnect.
> >
> > Here are Latency and Bandwidth benchmark results.
> > #---
> > # Benchmarking PingPong
> > # processes = 2
> > # map-by node
> > #---
> >
> > Hello, world.  I am 1 on node124
> > Hello, world.  I am 0 on node123
> > Size Latency (usec) Bandwidth (Mbps)
> > 11.654.62
> > 21.679.16
> > 41.6618.43
> > 81.6636.74
> > 161.8566.00
> > 321.83133.28
> > 641.83266.36
> > 1281.88519.10
> > 2561.99982.29
> > 5122.231752.37
> > 10242.583026.98
> > 20483.324710.76
> >
> > I read some of the FAQs and noted that OpenMPI prefers the faster
> available interconnect. In an effort to force it to use the gigabit
> interconnect I ran it as follows
> >
> > 1. mpirun -np 2 -machinefile machines -map-by node --mca btl tcp --mca
> btl_tcp_if_include em1 ./latency.ompi
> > 2. mpirun -np 2 -machinefile machines -map-by node --mca btl tcp,self,sm
> --mca btl_tcp_if_include em1 ./latency.ompi
> > 3. mpirun -np 2 -machinefile machines -map-by node --mca btl ^openib
> --mca btl_tcp_if_include em1 ./latency.ompi
> > 4. mpirun -np 2 -machinefile machines -map-by node --mca btl ^openib
> ./latency.ompi
> >
> > None of them resulted in a significantly different benchmark output.
> >
> > I am using OpenMPI by loading module on clustered environment and don't
> have admin access. It is configured for both TCP and OpenIB (confirmed from
> ompi_info). After trying all above mentioned methods without success I
> installed OpenMPI v1.8.2 in my home directory and disable openib with
> following configuration options
> >
> > --disable-openib-control-hdr-padding --disable-openib-dynamic-sl
> --disable-openib-connectx-xrc --disable-openib-udcm
> --disable-openib-rdmacm  --disable-btl-openib-malloc-alignment
> --disable-io-romio --without-openib --without-verbs
> >
> > Now, openib is not enabled (confirmed from ompi_info script) and there
> is no "openib.so" file in $prefix/lib/openmpi directory as well. Still,
> above mentioned mpirun commands are getting the same latency and bandwidth
> as that of InfiniBand.
> >
> > I tried mpirun in verbose mode with following command and here is the
> output
> >
> > Command:
> > mpirun -np 2 -machinefile machines -map-by node --mca btl tcp --mca
> btl_base_verbose 30 --mca btl_tcp_if_include em1 ./latency.ompi
> >
> > Output:
> > [node123.prv.sciama.cluster:88310] mca: base: components_register:
> registering btl components
> > [node123.prv.sciama.cluster:88310] mca: base: components_register: found
> loaded component tcp
> > [node123.prv.sciama.cluster:88310] mca: base: components_register:
> component tcp register function successful
> > [node123.prv.sciama.cluster:88310] mca: base: components_open: opening
> btl components
> > [node123.prv.sciama.cluster:88310] mca: base: components_open: found
> loaded component tcp
> > [node123.prv.sciama.cluster:88310] mca: base: components_open: component
> tcp open function successful
> > [node124.prv.sciama.cluster:90465] mca: base: components_register:
> registering btl components
> > [node124.prv.sciama.cluster:90465] mca: base: components_register: found
> loaded component tcp
> > [node124.prv.sciama.cluster:90465] mca: base: components_register:
> component tcp register function successful
> > [node124.prv.sciama.cluster:90465] mca: base: components_open: opening
> btl components
> > [node124.prv.sciama.cluster:90465] mca: base: components_open: found
> loaded component tcp
> > [node124.prv.sciama.cluster:90465] mca: base: components_open: component
> tcp open function successful
> > Hello, world.  I am 1 on node124
> > Hello, world.  I am 0 on node123
> > Size Latency(usec) Bandwidth(Mbps)
> > 14.18

[OMPI users] Multiple threads for an mpi process

[JR Cary]

This must be a very old topic.

I would like to run mpi with one process per node, e.g., using
-cpus-per-rank=1.  Then I want to use openmp inside of that.
But other times I will run with a rank on each physical core.

Inside my code I would like to detect which situation I am in.
Is there an openmpi api call to determine that?

Thanks.John

Re: [OMPI users] unaligned accesses

[Siegmar Gross]

Hi Jeff,

> We committed some fixes -- see if you can get farther with
> tonight's nightly tarball.

My small Java and C programs still break on Solaris 10 Sparc
and x86_64 and on Linux (Sun C 5.12) with different errors. I
have put everything into this email, because I have already
sent single error messages for these errors before for an
earlier version. It seems that nothing has changed for my
programs and my environment.


Java program:
=

tyr java 105 ompi_info | grep -e MPI: -e "C compiler:"
Open MPI: 1.9a1r32716
  C compiler: cc
tyr java 106 mpijavac InitFinalizeMain.java
warning: [path] bad path element 
"/usr/local/openmpi-1.9_64_cc/lib64/shmem.jar": no such file or directory
1 warning
tyr java 107 mpiexec -np 1 java InitFinalizeMain
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7ea3c7f0, pid=21678, tid=2
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build 1.8.0-b132)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode solaris-sparc 
compressed oops)
# Problematic frame:
# C  [libc.so.1+0x3c7f0]  strlen+0x50
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" 
before starting Java again
#
# An error report file with more information is saved as:
# /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid21678.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--
mpiexec noticed that process rank 0 with PID 0 on node tyr exited on signal 6 
(Abort).
--
tyr java 108 




tyr java 108 /usr/local/gdb-7.6.1_64_gcc/bin/gdb 
/usr/local/openmpi-1.9_64_cc/bin/mpiexec
GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.10".
For bug reporting instructions, please see:
...
Reading symbols from 
/export2/prog/SunOS_sparc/openmpi-1.9_64_cc/bin/orterun...done.
(gdb) run -np 1 java InitFinalizeMain 
Starting program: /usr/local/openmpi-1.9_64_cc/bin/mpiexec -np 1 java 
InitFinalizeMain
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
[New LWP2]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7ea3c7f0, pid=21696, tid=2
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build 1.8.0-b132)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode solaris-sparc 
compressed oops)
# Problematic frame:
# C  [libc.so.1+0x3c7f0]  strlen+0x50
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" 
before starting Java again
#
# An error report file with more information is saved as:
# /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid21696.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--
mpiexec noticed that process rank 0 with PID 0 on node tyr exited on signal 6 
(Abort).
--
[LWP2 exited]
[New Thread 2]
[Switching to Thread 1 (LWP 1)]
sol_thread_fetch_registers: td_ta_map_id2thr: no thread can be found to satisfy 
query
(gdb) bt
#0  0x7f6173d0 in rtld_db_dlactivity () from /usr/lib/sparcv9/ld.so.1
#1  0x7f6175a8 in rd_event () from /usr/lib/sparcv9/ld.so.1
#2  0x7f618950 in lm_delete () from /usr/lib/sparcv9/ld.so.1
#3  0x7f6226bc in remove_so () from /usr/lib/sparcv9/ld.so.1
#4  0x7f624574 in remove_hdl () from /usr/lib/sparcv9/ld.so.1
#5  0x7f61d97c in dlclose_core () from /usr/lib/sparcv9/ld.so.1
#6  0x7f61d9d4 in dlclose_intn () from /usr/lib/sparcv9/ld.so.1
#7  0x7f61db0c in dlclose () from /usr/lib/sparcv9/ld.so.1
#8  0x7e4e6f88 in vm_close ()
   from /usr/local/openmpi-1.9_64_cc/lib64/libopen-pal.so.0
#9  0x7e4e4274 in lt_dlclose ()
   from /usr/local/openmpi-1.9_64_cc/lib64/libopen-pal.so.0
#10 0x7e53a574 in ri_destructor (obj=0x0)
at 
../../../../openmpi-1.9a1r32716/opal/mca/base/mca_base_component_repository.c:382
#11 0x

Re: [OMPI users] Multiple threads for an mpi process

[Tim Prince]

On 9/12/2014 6:14 AM, JR Cary wrote:

This must be a very old topic.

I would like to run mpi with one process per node, e.g., using
-cpus-per-rank=1.  Then I want to use openmp inside of that.
But other times I will run with a rank on each physical core.

Inside my code I would like to detect which situation I am in.
Is there an openmpi api call to determine that?

omp_get_num_threads() should work.  Unless you want to choose a 
different non-parallel algorithm for this case, a single thread omp 
parallel region works fine.
You should soon encounter cases where you want intermediate choices, 
such as 1 rank per CPU package and 1 thread per core, even if you stay 
away from platforms with more than 12 cores per CPU.

Re: [OMPI users] Multiple threads for an mpi process

[JR Cary]

On 9/12/14, 7:27 AM, Tim Prince wrote:


On 9/12/2014 6:14 AM, JR Cary wrote:

This must be a very old topic.

I would like to run mpi with one process per node, e.g., using
-cpus-per-rank=1.  Then I want to use openmp inside of that.
But other times I will run with a rank on each physical core.

Inside my code I would like to detect which situation I am in.
Is there an openmpi api call to determine that?

omp_get_num_threads() should work.  Unless you want to choose a 
different non-parallel algorithm for this case, a single thread omp 
parallel region works fine.
You should soon encounter cases where you want intermediate choices, 
such as 1 rank per CPU package and 1 thread per core, even if you stay 
away from platforms with more than 12 cores per CPU.


I may not understand, so I will try to ask in more detail.

Suppose I am running on a four-core processor (and my code likes one 
thread per core).


In case 1 I do

  mpiexec -np 2 myexec

and I want to know that each mpi process should use 2 threads.

If instead I did

  mpiexec -np 4 myexec

I want to know that each mpi process should use one thread.

Will omp_get_num_threads() should return a different value for those two 
cases?


Perhaps I am not invoking mpiexec correctly.
I use MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, 
&threadSupport), and regardless

of what how I invoke mpiexec (-n 1, -n 2, -n 4), I see 2 openmp processes
and 1 openmp threads (have not called omp_set_num_threads).
When I run serial, I see 8 openmp processes and 1 openmp threads.
So I must be missing an arg to mpiexec?

This is a 4-core haswell with hyperthreading to get 8.



Thx.

Re: [OMPI users] Multiple threads for an mpi process

[Ralph Castain]

Hmmm...well, the info is there. There is an envar OMPI_COMM_WORLD_LOCAL_SIZE 
which tells you how many procs are on this node. If you tell your proc how many 
cores (or hwthreads) to use, it would be a simple division to get what you want.

You could also detect the number of cores or hwthreads via a call to hwloc, but 
I don't know if you want to link that deep and MPI doesn't have a function for 
that purpose. Could be that OpenMP provides a call for that purpose?


On Sep 12, 2014, at 7:22 AM, JR Cary  wrote:

> 
> 
> On 9/12/14, 7:27 AM, Tim Prince wrote:
>> 
>> On 9/12/2014 6:14 AM, JR Cary wrote:
>>> This must be a very old topic.
>>> 
>>> I would like to run mpi with one process per node, e.g., using
>>> -cpus-per-rank=1.  Then I want to use openmp inside of that.
>>> But other times I will run with a rank on each physical core.
>>> 
>>> Inside my code I would like to detect which situation I am in.
>>> Is there an openmpi api call to determine that?
>>> 
>> omp_get_num_threads() should work.  Unless you want to choose a different 
>> non-parallel algorithm for this case, a single thread omp parallel region 
>> works fine.
>> You should soon encounter cases where you want intermediate choices, such as 
>> 1 rank per CPU package and 1 thread per core, even if you stay away from 
>> platforms with more than 12 cores per CPU.
> 
> I may not understand, so I will try to ask in more detail.
> 
> Suppose I am running on a four-core processor (and my code likes one thread 
> per core).
> 
> In case 1 I do
> 
>   mpiexec -np 2 myexec
> 
> and I want to know that each mpi process should use 2 threads.
> 
> If instead I did
> 
>   mpiexec -np 4 myexec
> 
> I want to know that each mpi process should use one thread.
> 
> Will omp_get_num_threads() should return a different value for those two 
> cases?
> 
> Perhaps I am not invoking mpiexec correctly. 
> I use MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &threadSupport), and 
> regardless
> of what how I invoke mpiexec (-n 1, -n 2, -n 4), I see 2 openmp processes 
> and 1 openmp threads (have not called omp_set_num_threads).
> When I run serial, I see 8 openmp processes and 1 openmp threads.
> So I must be missing an arg to mpiexec?
> 
> This is a 4-core haswell with hyperthreading to get 8.
> 
>   
> 
> Thx.
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/09/25322.php

Re: [OMPI users] Multiple threads for an mpi process

[Tim Prince]

On 9/12/2014 9:22 AM, JR Cary wrote:



On 9/12/14, 7:27 AM, Tim Prince wrote:


On 9/12/2014 6:14 AM, JR Cary wrote:

This must be a very old topic.

I would like to run mpi with one process per node, e.g., using
-cpus-per-rank=1.  Then I want to use openmp inside of that.
But other times I will run with a rank on each physical core.

Inside my code I would like to detect which situation I am in.
Is there an openmpi api call to determine that?

omp_get_num_threads() should work.  Unless you want to choose a 
different non-parallel algorithm for this case, a single thread omp 
parallel region works fine.
You should soon encounter cases where you want intermediate choices, 
such as 1 rank per CPU package and 1 thread per core, even if you 
stay away from platforms with more than 12 cores per CPU.


I may not understand, so I will try to ask in more detail.

Suppose I am running on a four-core processor (and my code likes one 
thread per core).


In case 1 I do

  mpiexec -np 2 myexec

and I want to know that each mpi process should use 2 threads.

If instead I did

  mpiexec -np 4 myexec

I want to know that each mpi process should use one thread.

Will omp_get_num_threads() should return a different value for those 
two cases?


Perhaps I am not invoking mpiexec correctly.
I use MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, 
&threadSupport), and regardless

of what how I invoke mpiexec (-n 1, -n 2, -n 4), I see 2 openmp processes
and 1 openmp threads (have not called omp_set_num_threads).
When I run serial, I see 8 openmp processes and 1 openmp threads.
So I must be missing an arg to mpiexec?

This is a 4-core haswell with hyperthreading to get 8.


Sorry, I assumed you were setting OMP_NUM_THREADS for your runs.  If you 
don't do that, each instance of OpenMP will try to run 8 threads, where 
you probably want just 1 thread per core.  I turn off hyperthreading in 
BIOS on my machines, as I never run anything which would benefit from it.

[OMPI users] more info on SIGSEGV for Java and openmpi-1.9a1r32716 on Solaris

[Siegmar Gross]

Hi,

is the following info helpful?

tyr java 160 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.10".
For bug reporting instructions, please see:
...
Reading symbols from 
/export2/prog/SunOS_sparc/openmpi-1.9_64_cc/bin/orterun...done.
(gdb) run -np 1 java InitFinalizeMain 
Starting program: /usr/local/openmpi-1.9_64_cc/bin/mpiexec -np 1 java 
InitFinalizeMain
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
[New LWP2]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7ea3c7f0, pid=22871, tid=2
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build 1.8.0-b132)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode solaris-sparc 
compressed oops)
# Problematic frame:
# C  [libc.so.1+0x3c7f0]  strlen+0x50
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid22871.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--
mpiexec noticed that process rank 0 with PID 0 on node tyr exited on signal 6 
(Abort).
--
[LWP2 exited]
[New Thread 2]
[Switching to Thread 1 (LWP 1)]
sol_thread_fetch_registers: td_ta_map_id2thr: no thread can be found to satisfy 
query
(gdb) bt
#0  0x7f6173d0 in rtld_db_dlactivity () from /usr/lib/sparcv9/ld.so.1
#1  0x7f6175a8 in rd_event () from /usr/lib/sparcv9/ld.so.1
#2  0x7f618950 in lm_delete () from /usr/lib/sparcv9/ld.so.1
#3  0x7f6226bc in remove_so () from /usr/lib/sparcv9/ld.so.1
#4  0x7f624574 in remove_hdl () from /usr/lib/sparcv9/ld.so.1
#5  0x7f61d97c in dlclose_core () from /usr/lib/sparcv9/ld.so.1
#6  0x7f61d9d4 in dlclose_intn () from /usr/lib/sparcv9/ld.so.1
#7  0x7f61db0c in dlclose () from /usr/lib/sparcv9/ld.so.1
#8  0x7e4e6f88 in vm_close () from 
/usr/local/openmpi-1.9_64_cc/lib64/libopen-pal.so.0
#9  0x7e4e4274 in lt_dlclose () from 
/usr/local/openmpi-1.9_64_cc/lib64/libopen-pal.so.0
#10 0x7e53a574 in ri_destructor (obj=0x0) at 
../../../../openmpi-1.9a1r32716/opal/mca/base/mca_base_component_repository.c:382
#11 0x7e537d50 in opal_obj_run_destructors (object=0x0) at 
../../../../openmpi-1.9a1r32716/opal/class/opal_object.h:446
#12 0x7e539de4 in mca_base_component_repository_release 
(component=0xf000)
at 
../../../../openmpi-1.9a1r32716/opal/mca/base/mca_base_component_repository.c:240
#13 0x7e540448 in mca_base_component_unload (component=0x0, 
output_id=-2145509376)
at 
../../../../openmpi-1.9a1r32716/opal/mca/base/mca_base_components_close.c:47
#14 0x7e5404ec in mca_base_component_close 
(component=0xff7b30ff, output_id=255)
at 
../../../../openmpi-1.9a1r32716/opal/mca/base/mca_base_components_close.c:60
#15 0x7e5405fc in mca_base_components_close (output_id=767, 
components=0x0, skip=0xff7f73cdf800)
at 
../../../../openmpi-1.9a1r32716/opal/mca/base/mca_base_components_close.c:86
#16 0x7e54053c in mca_base_framework_components_close (framework=0xff, 
skip=0xff7c801c4000)
at 
../../../../openmpi-1.9a1r32716/opal/mca/base/mca_base_components_close.c:68
#17 0x7ee48d68 in orte_oob_base_close () at 
../../../../openmpi-1.9a1r32716/orte/mca/oob/base/oob_base_frame.c:98
#18 0x7e56c23c in mca_base_framework_close 
(framework=0xff7e4e413cff)
at ../../../../openmpi-1.9a1r32716/opal/mca/base/mca_base_framework.c:187
#19 0x7bb13f00 in rte_finalize () at 
../../../../../openmpi-1.9a1r32716/orte/mca/ess/hnp/ess_hnp_module.c:857
#20 0x7ec3adf0 in orte_finalize () at 
../../openmpi-1.9a1r32716/orte/runtime/orte_finalize.c:66
#21 0x0001e264 in orterun (argc=4607, argv=0x0) at 
../../../../openmpi-1.9a1r32716/orte/tools/orterun/orterun.c:1099
#22 0x000146d4 in main (argc=255, argv=0xff7f0af87800) at 
../../../../openmpi-1.9a1r32716/orte/tools/orterun/main.c:13
(gdb) thread 1
[Switching to thread 1 (LWP1)]
#0  0x7f6173d0 in rtld_db_dlactivity () from /usr/lib/sparcv9/ld.so.1

[OMPI users] about using mpi-thread-multiple

[etcamargo]

Hi,

I would like to know what is the mpi version recomended for running 
multiple mpi call per process, i.e., MPI_THREAD_MULTIPLE in 
MPI_Init_thread();



Thanks,

Edson