I would try using mpiexec or mpirun instead of srun to launch the job. Those commands are part of openmpi – they should be slurm-aware and launch the job in a way that initializes MPI properly.
-----Original Message----- From: "[email protected]" <[email protected]> Reply-To: slurm-dev <[email protected]> Date: Friday, January 20, 2017 at 8:16 AM To: slurm-dev <[email protected]> Subject: [slurm-dev] slurm-dev srun openmpi errors Hello everyone, i'm using slurm 15.08.07 and openmpi 1.6.5 on both my master and my nodes. Whenever i try the command "srun -n2 ./miniFE.x", i get the following ORTE Error Messages: [odroid-c1-2:10790] [[300,1],1] ORTE_ERROR_LOG: A message is attempting to be sent to a process whose contact information is unknown in file rml_oob_send.c at line 145 [odroid-c1-2:10790] [[300,1],1] attempted to send to [[300,1],0]: tag 15 [odroid-c1-2:10790] [[300,1],1] ORTE_ERROR_LOG: A message is attempting to be sent to a process whose contact information is unknown in file grpcomm_hier_module.c at line 355 [odroid-c1-2:10790] [[300,1],1] ORTE_ERROR_LOG: A message is attempting to be sent to a process whose contact information is unknown in file base/grpcomm_base_modex.c at line 469 [odroid-c1-2:10790] [[300,1],1] ORTE_ERROR_LOG: A message is attempting to be sent to a process whose contact information is unknown in file grpcomm_hier_module.c at line 476 -------------------------------------------------------------------------- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_grpcomm_modex failed --> Returned "A message is attempting to be sent to a process whose contact information is unknown" (-117) instead of "Success" (0) -------------------------------------------------------------------------- [odroid-c1-2:10790] *** An error occurred in MPI_Init [odroid-c1-2:10790] *** on a NULL communicator [odroid-c1-2:10790] *** Unknown error [odroid-c1-2:10790] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort -------------------------------------------------------------------------- An MPI process is aborting at a time when it cannot guarantee that all of its peer processes in the job will be killed properly. You should double check that everything has shut down cleanly. Reason: Before MPI_INIT completed Local host: odroid-c1-2 PID: 10790 -------------------------------------------------------------------------- srun: error: odroid-c1-2: task 1: Exited with exit code 1 ^Csrun: interrupt (one more within 1 sec to abort) srun: task 0: running srun: task 1: exited abnormally odroid-c1-2 is my target node. My Master is running Centos7 and the odroid Ubuntu14.04. OMPI was built with the same config-flags (especially --with-slurm) on both machines. If i try "srun -n1 ./miniFE.x" it works fine. Same when i execute "mpirun -n 2 -host odroid-c1-2 ./miniFE.x". Any advice on how to fix this? Regards, M. Wagner
