Ralph,
For the following two different commandline invocations of the LAPACK benchmark
$MPI_DIR/bin/mpirun -np $NSLOTS --report-bindings --hostfile hostfile-no_slots
--mca btl_tcp_if_include eth0 --hetero-nodes --use-hwthread-cpus --bind-to
hwthread --prefix $MPI_DIR $BENCH_DIR/$APP_DIR/$APP_BIN
$MPI_DIR/bin/mpirun -np $NSLOTS --report-bindings --hostfile hostfile-no_slots
--mca btl_tcp_if_include eth0 --hetero-nodes --bind-to-core --prefix $MPI_DIR
$BENCH_DIR/$APP_DIR/$APP_BIN
I'm receiving the same kinds of OpenMPI error messages (but for different nodes
in the ring):
[csclprd3-0-16:25940] *** Process received signal ***
[csclprd3-0-16:25940] Signal: Bus error (7)
[csclprd3-0-16:25940] Signal code: Non-existant physical address (2)
[csclprd3-0-16:25940] Failing at address: 0x7f8b1b5a2600
--------------------------------------------------------------------------
mpirun noticed that process rank 82 with PID 25936 on node
csclprd3-0-16 exited on signal 7 (Bus error).
--------------------------------------------------------------------------
16 total processes killed (some possibly by mpirun during cleanup)
It seems to occur on systems that have more than one, physical CPU installed.
Could
this be due to a lack of the correct NUMA libraries being installed?
-Bill L.
________________________________
From: users [users-boun...@open-mpi.org] on behalf of Ralph Castain
[r...@open-mpi.org]
Sent: Sunday, April 05, 2015 6:09 PM
To: Open MPI Users
Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3
On Apr 5, 2015, at 5:58 PM, Lane, William
<william.l...@cshs.org<mailto:william.l...@cshs.org>> wrote:
I think some of the Intel Blade systems in the cluster are
dual core, but don't support hyperthreading. Maybe it
would be better to exclude hyperthreading altogether
from submitted OpenMPI jobs?
Yes - or you can add "--hetero-nodes -use-hwthread-cpus --bind-to hwthread" to
the cmd line. This tells mpirun that the nodes aren't all the same, and so it
has to look at each node's topology instead of taking the first node as the
template for everything. The second tells it to use the HTs as independent cpus
where they are supported.
I'm not entirely sure the suggestion will work - if we hit a place where HT
isn't supported, we may balk at being asked to bind to HTs. I can probably make
a change that supports this kind of hetero arrangement (perhaps something like
bind-to pu) - might make it into 1.8.5 (we are just starting the release
process on it now).
OpenMPI doesn't crash, but it doesn't run the LAPACK
benchmark either.
Thanks again Ralph.
Bill L.
________________________________
From: users [users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org>] on
behalf of Ralph Castain [r...@open-mpi.org<mailto:r...@open-mpi.org>]
Sent: Wednesday, April 01, 2015 8:40 AM
To: Open MPI Users
Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3
Bingo - you said the magic word. This is a terminology issue. When we say
"core", we mean the old definition of "core", not "hyperthreads". If you want
to use HTs as your base processing unit and bind to them, then you need to
specify --bind-to hwthread. That warning should then go away.
We don't require a swap region be mounted - I didn't see anything in your
original message indicating that OMPI had actually crashed, but just wasn't
launching due to the above issue. Were you actually seeing crashes as well?
On Wed, Apr 1, 2015 at 8:31 AM, Lane, William
<william.l...@cshs.org<mailto:william.l...@cshs.org>> wrote:
Ralph,
Here's the associated hostfile:
#openMPI hostfile for csclprd3
#max slots prevents oversubscribing csclprd3-0-9
csclprd3-0-0 slots=12 max-slots=12
csclprd3-0-1 slots=6 max-slots=6
csclprd3-0-2 slots=6 max-slots=6
csclprd3-0-3 slots=6 max-slots=6
csclprd3-0-4 slots=6 max-slots=6
csclprd3-0-5 slots=6 max-slots=6
csclprd3-0-6 slots=6 max-slots=6
csclprd3-0-7 slots=32 max-slots=32
csclprd3-0-8 slots=32 max-slots=32
csclprd3-0-9 slots=32 max-slots=32
csclprd3-0-10 slots=32 max-slots=32
csclprd3-0-11 slots=32 max-slots=32
csclprd3-0-12 slots=12 max-slots=12
csclprd3-0-13 slots=24 max-slots=24
csclprd3-0-14 slots=16 max-slots=16
csclprd3-0-15 slots=16 max-slots=16
csclprd3-0-16 slots=24 max-slots=24
csclprd3-0-17 slots=24 max-slots=24
csclprd3-6-1 slots=4 max-slots=4
csclprd3-6-5 slots=4 max-slots=4
The number of slots also includes hyperthreading
cores.
One more question, would not having defined swap
partitions on all the nodes in the ring cause OpenMPI
to crash? Because no swap partitions are defined
for any of the above systems.
-Bill L.
________________________________
From: users [users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org>] on
behalf of Ralph Castain [r...@open-mpi.org<mailto:r...@open-mpi.org>]
Sent: Wednesday, April 01, 2015 5:04 AM
To: Open MPI Users
Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3
The warning about binding to memory is due to not having numactl-devel
installed on the system. The job would still run, but we are warning you that
we cannot bind memory to the same domain as the core where we bind the process.
Can cause poor performance, but not fatal. I forget the name of the param, but
you can tell us to "shut up" :-)
The other warning/error indicates that we aren't seeing enough cores on the
allocation you gave us via the hostile to support one proc/core - i.e., we
didn't at least 128 cores in the sum of the nodes you told us about. I take it
you were expecting that there were that many or more?
Ralph
On Wed, Apr 1, 2015 at 12:54 AM, Lane, William
<william.l...@cshs.org<mailto:william.l...@cshs.org>> wrote:
I'm having problems running OpenMPI jobs
(using a hostfile) on an HPC cluster running
ROCKS on CentOS 6.3. I'm running OpenMPI
outside of Sun Grid Engine (i.e. it is not submitted
as a job to SGE). The program being run is a LAPACK
benchmark. The commandline parameter I'm
using to run the jobs is:
$MPI_DIR/bin/mpirun -np $NSLOTS -bind-to-core -report-bindings --hostfile
hostfile --mca btl_tcp_if_include eth0 --prefix $MPI_DIR
$BENCH_DIR/$APP_DIR/$APP_BIN
Where MPI_DIR=/hpc/apps/mpi/openmpi/1.8.2/
NSLOTS=128
I'm getting errors of the form and OpenMPI never runs the LAPACK benchmark:
--------------------------------------------------------------------------
WARNING: a request was made to bind a process. While the system
supports binding the process itself, at least one node does NOT
support binding memory to the process location.
Node: csclprd3-0-11
This usually is due to not having the required NUMA support installed
on the node. In some Linux distributions, the required support is
contained in the libnumactl and libnumactl-devel packages.
This is a warning only; your job will continue, though performance may be
degraded.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
A request was made to bind to that would result in binding more
processes than cpus on a resource:
Bind to: CORE
Node: csclprd3-0-11
#processes: 2
#cpus: 1
You can override this protection by adding the "overload-allowed"
option to your binding directive.
--------------------------------------------------------------------------
The only installed numa packages are:
numactl.x86_64 2.0.7-3.el6
@centos6.3-x86_64-0/$
When I search for the available NUMA packages I find:
yum search numa | less
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
============================== N/S Matched: numa
===============================
numactl-devel.i686 : Development package for building Applications that
use numa
numactl-devel.x86_64 : Development package for building Applications
that use
: numa
numad.x86_64 : NUMA user daemon
numactl.i686 : Library for tuning for Non Uniform Memory Access machines
numactl.x86_64 : Library for tuning for Non Uniform Memory Access
machines
Do I need to install additional and/or different NUMA packages in order to get
OpenMPI to work
on this cluster?
-Bill Lane
IMPORTANT WARNING: This message is intended for the use of the person or entity
to which it is addressed and may contain information that is privileged and
confidential, the disclosure of which is governed by applicable law. If the
reader of this message is not the intended recipient, or the employee or agent
responsible for delivering it to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this information is
strictly prohibited. Thank you for your cooperation.
_______________________________________________
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Searchable archives:
http://www.open-mpi.org/community/lists/users/2015/04/index.php
IMPORTANT WARNING: This message is intended for the use of the person or entity
to which it is addressed and may contain information that is privileged and
confidential, the disclosure of which is governed by applicable law. If the
reader of this message is not the intended recipient, or the employee or agent
responsible for delivering it to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this information is
strictly prohibited. Thank you for your cooperation.
_______________________________________________
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/04/26589.php
IMPORTANT WARNING: This message is intended for the use of the person or entity
to which it is addressed and may contain information that is privileged and
confidential, the disclosure of which is governed by applicable law. If the
reader of this message is not the intended recipient, or the employee or agent
responsible for delivering it to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this information is
strictly prohibited. Thank you for your cooperation.
_______________________________________________
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/04/26611.php
IMPORTANT WARNING: This message is intended for the use of the person or entity
to which it is addressed and may contain information that is privileged and
confidential, the disclosure of which is governed by applicable law. If the
reader of this message is not the intended recipient, or the employee or agent
responsible for delivering it to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this information is
strictly prohibited. Thank you for your cooperation.
