We are trying SLURM for the first time, and prior to this I've always built
OMPI with Torque support. I was hoping that someone with more experience
than I with both OMPI and SLURM might provide a bit of up-front advice.
My situation is that we are running CentOS 7.3 (soon to be 7.4), we use
Mellanox cards of several generations, but my systems team tells me the
driver version is the same everywhere.
82:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0
5GT/s - IB QDR / 10GigE] (rev b0)
16:00.0 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3]
We have mixed NFSv3 shared directories and a Lustre filesystem (DDN).
In the past, we had issues with using `dlopen` and we've had much grief
with OMPI placing jobs on processors correctly, we think because we used
cpusets at one point and use cgroups now and jobs share nodes with other
jobs. My previous build options were
export CONFIGURE_FLAGS='--disable-dlopen --enable-shared'
export COMPILERS='CC=gcc CXX=g++ FC=gfortran F77=gfortran'
export COMP_NAME='gcc-4.8.5'
export PREFIX=/shared/nfs/directory
./configure \
--prefix=${PREFIX} \
--mandir=${PREFIX}/share/man \
--with-tm \
--with-verbs \
$CONFIGURE_FLAGS \
$COMPILERS
Additionally, we have typically included the following lines in our
$PREFIX/etc/openmpi-mca-params.conf
orte_hetero_nodes=1
hwloc_base_binding_policy=none
Those may be there for purely historical reasons. So far as I know, there
is no deterministic test recorded anywhere that would detect whether those
are still needed.
For this new resource manager, I am thinking that the compiler flags stay
the same, but the configure be changed to
./configure \
--prefix=${PREFIX} \
--mandir=${PREFIX}/share/man \
--with-slurm \
--with-pmi=/usr/include/slurm \
--with-verbs \
$CONFIGURE_FLAGS \
$COMPILERS
I am curious, what file system support does --lustre-support enable?
I will be installing three versions of OMPI to start: 1.10.7, 2.1.2, and
3.0.0. Are there changes to the configure line that are a priori known to
be needed.
There are references on the FAQ and other installation notes that lead me
to believe they are a bit out of date, so I am asking preemptively here.
Apologies if that is an incorrect assessment.
Thanks, -- bennet
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
