Hi! I'm working on a system with KNL and OmniPath and I'm trying to launch a job but it fails. Could someone please advise what parameters I need to add to make it work properly? At first I need to make it work within one node, however later I need to use multiple nodes and eventually I may need to switch to TCP to run a hybrid job where some nodes are connected via Infiniband and some nodes are connected via OmniPath.
So far without any extra parameters I get: ``` By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: XXXXXX Local adapter: hfi1_0 Local port: 1 ``` If I add `OMPI_MCA_btl_openib_allow_ib="true"` then I get: ``` Error obtaining unique transport key from ORTE (orte_precondition_transports not present in the environment). Local host: XXXXXX ``` Then I tried adding OMPI_MCA_mtl="psm2" or OMPI_MCA_mtl="ofi" to make it use omnipath or OMPI_MCA_btl="sm,self" to make it use only shared memory. But these parameters did not make any difference. There does not seem to be much omni-path related documentation, at least I was not able to find anything that would help me but perhaps I missed something: https://www.open-mpi.org/faq/?category=running#opa-support https://www.open-mpi.org/faq/?category=opa This is the `configure` line: ``` ./configure --prefix=XXXXX --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --enable-shared --with-hwloc=$EBROOTHWLOC --with-psm2 --with-libevent=$EBROOTLIBEVENT --without-orte --disable-oshmem --with-cuda=$EBROOTCUDA --with-gpfs --with-slurm --with-pmix=external --with-libevent=external --with-ompi-pmix-rte ``` Which also raises another question: if it was built with `--without-orte` then why do I get an error about failing to get something from ORTE. The OpenMPI version is `4.1.0rc1` built with `gcc-9.3.0`. Thank you in advance! Regards, Pavel Mezentsev.
