[slurm-dev] scancel cannot successfully cancel the mpi job create by running salloc

[10000]

Hi, 
    I meet a weird problem. I am "salloc mpirun" to start a mpi task on Slurm 
cluster. And then, if I cancel the job by running "scancel <jobid>" and after 
waiting about 30 seconds, there prints "srun: error: HK-IDC1-10-1-72-101: task 
0: Killed". But I find that it only kill the first process that mpi creates and 
doesn't kill the other processes including the "mpirun" and "salloc". Then all 
the process are just like been frozen but the status providing by "ps" has show 
that these processes are in status "Rsl". Then I decide to kill one of the rest 
mpi processes to trigger the mpi failure for cleanning up the rest mpi 
processes, I find that is not worked (That should be work since our process are 
always doing ALL_REDUCE.). Then if I send Ctrl-C to (twice) the shell running 
"salloc", i will kill "salloc" and "mpirun" but leave the rest processes that 
mpirun created. So I have to kill them one by one at every node where I just 
running it. However, if I cancel the job by send Ctrl-C at the shell running 
"salloc" at first it do work and clean up all processes.
    Here are the command I run salloc:
    salloc -n12 --ntasks-per-node=4 /usr/local/mvapich2-2.2rc1/bin/mpirun ./aa
    
    The code of aa are following:
    
#include <stdio.h>
#include <unistd.h>
#include <mpi.h>
int main(int argc,char **argv)
{
    int rank,size;
    double in=1.0,out;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    while (1)
    {
        MPI_Barrier(MPI_COMM_WORLD);
        MPI_Allreduce(&in, &out, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);
        printf("Running! %lf\n",out);
        sleep(1);
    }
    MPI_Finalize();
    return 0;
}

      The MPI we using are MVAPICH2-2.2rc1 which is the latest version of 
mvapich.
      The SLURM we using are slurm-15.08.11.tar.bz2 which is also the latest 
version of slurm15

Any one have an idea?

Thanks
Yingdi Guo