Hi, I’m having an issue with srun's --no-alloc flag with mpi which I can reproduce with a fairly simple example. When I run a simple one-core mpi test program as “slurmUser” (the account that has the --no-alloc privilege) it succeeds:
srun -p psfehq -n 1 -o logs/test.log -w psana1507 python ~/ipsana/mpi_simpletest.py However when I add the --no-alloc flag it fails in a way that appears to break mpi (see logfile output and other slurm/mpi info below). It fails similarly on 2 cores. srun --no-alloc -p psfehq -n 1 -o logs/test.log -w psana1507 python ~/ipsana/mpi_simpletest.py srun: do not allocate resources srun: error: psana1507: task 0: Exited with exit code 1 Would anyone have any suggestions for how I could make the “--no-alloc” flag work with mpi? Thanks! chris ------------------------------------------------------------------------------------------------------ Logfile error with --no-alloc flag: (ana-4.0.12) psanagpu105:batchtest_slurm$ more logs/test.log -------------------------------------------------------------------------- The application appears to have been direct launched using "srun", but OMPI was not built with SLURM support. This usually happens when OMPI was not configured --with-slurm and we weren't able to discover a SLURM installation in the usual places. Please configure as appropriate and try again. -------------------------------------------------------------------------- *** An error occurred in MPI_Init_thread *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [psana1507:13884] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! (ana-4.0.12) psanagpu105:batchtest_slurm$ System information: (ana-4.0.12) psanagpu105:batchtest_slurm$ conda list | grep mpi mpi 1.0 openmpi conda-forge mpi4py 3.0.3 py27h9ab638b_1 conda-forge openmpi 4.1.0 h9b22176_1 conda-forge (ana-4.0.12) psanagpu105:batchtest_slurm$ srun --mpi=list srun: MPI types are... srun: cray_shasta srun: none srun: pmi2 srun: pmix srun: pmix_v3 (ana-4.0.12) psanagpu105:batchtest_slurm$ srun --version slurm 20.11.3 (ana-4.0.12) psanagpu105:batchtest_slurm$
