I have seen the clocks out of sync. I use a MPI program that checks the
local time after syncing on a barrier.
I also use:
bgilmer@jupiter:~> srun -t 1 -N90 date +%s | sort -n | uniq -c
90 1484229062
to see if there is a spread. On a small job like this there should not be
any variance. As the number of tasks increase the you should begin to see a
spread.
bgilmer@jupiter:~> srun -t 1 -n 1024 date +%s | sort -n | uniq -c
1 1484229252
343 1484229253
278 1484229254
232 1484229255
170 1484229256
This was run on a small Cray XC.
Hope this was helpful.
On Wed, Jan 11, 2017 at 1:22 PM, Pritchard Jr., Howard <[email protected]>
wrote:
> Hi SLURM folks,
>
> I recently got SLURM (16.05.6) set up on a small cluster (48 nodes x86_64
> + Intel OPA)
> and things appear to be nominal except for one odd performance problem
> as far as srun launch times go. I don’t observe this on other clusters
> running
> SLURM at our site.
>
> What I’m observing is that regardless of whether or not the application
> being
> launched is a command (e.g. /bin/hostname) or an MPI application, I get
> reasonable
> job launch times when using one node, but as soon as I use two or
> morenodes, there
> is about a 10 second overhead to get the processes on the additional nodes
> started:
>
> For example:
>
> [hpp@hi-master ~]$ srun -n 8 -N 1 date
>
> Wed Jan 11 12:11:29 MST 2017
>
> Wed Jan 11 12:11:29 MST 2017
>
> Wed Jan 11 12:11:29 MST 2017
>
> Wed Jan 11 12:11:29 MST 2017
>
> Wed Jan 11 12:11:29 MST 2017
>
> Wed Jan 11 12:11:29 MST 2017
>
> Wed Jan 11 12:11:29 MST 2017
>
> Wed Jan 11 12:11:29 MST 2017
>
> [hpp@hi-master ~]$ srun -n 8 -N 2 date
>
> Wed Jan 11 12:10:35 MST 2017
>
> Wed Jan 11 12:10:35 MST 2017
>
> Wed Jan 11 12:10:35 MST 2017
>
> Wed Jan 11 12:10:35 MST 2017
>
> Wed Jan 11 12:10:44 MST 2017
>
> Wed Jan 11 12:10:44 MST 2017
>
> Wed Jan 11 12:10:44 MST 2017
>
> Wed Jan 11 12:10:44 MST 2017
>
> [hpp@hi-master ~]$ srun -n 8 -N 4 date
>
> Wed Jan 11 12:10:57 MST 2017
>
> Wed Jan 11 12:10:57 MST 2017
>
> Wed Jan 11 12:11:07 MST 2017
>
> Wed Jan 11 12:11:06 MST 2017
>
> Wed Jan 11 12:11:07 MST 2017
>
> Wed Jan 11 12:11:06 MST 2017
>
> Wed Jan 11 12:11:07 MST 2017
>
> Wed Jan 11 12:11:07 MST 2017
>
>
> Anyone observed this problem before?
>
>
> Any suggestions on how to resolve this problem would be much
>
> appreciated.
>
>
> Thanks,
>
>
> Howard
>
>
> --
> Howard Pritchard
> HPC-DES
> Los Alamos National Laboratory
>
>
--
Speak when you are angry--and you will make the best speech you'll ever
regret.
- Laurence J. Peter
/*
Everyone's hello world
This one displays the current time.
To build:
sphinx:~ # module load PrgEnv-gnu
sphinx:~ # module load craype-haswell
sphinx:~ # module load cray-mpich
sphinx:~ # CC tsync.c
To run:
srun -N4 ./a.out %s
*/
#include "mpi.h"
#include <time.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
int rank, nprocs;
char outstr[200];
time_t t;
struct tm *tmp;
char host[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Init(&argc,&argv);
// Get the number of processors
MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
// Get the rank
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
// Get the name of the processor
MPI_Get_processor_name(host, &name_len);
MPI_Barrier(MPI_COMM_WORLD);
t = time(NULL);
tmp = localtime(&t);
if (tmp == NULL) {
perror("localtime");
MPI_Finalize();
exit(EXIT_FAILURE);
}
if (strftime(outstr, sizeof(outstr), argv[1], tmp) == 0) {
fprintf(stderr, "strftime returned 0");
MPI_Finalize();
exit(EXIT_FAILURE);
}
printf("%s %s %d %d\n", outstr, host, rank, nprocs);
MPI_Finalize();
exit(EXIT_SUCCESS);
}
