On Tue, Apr 23, 2013 at 12:21:49PM +0400, ???????????????????? ???????????? wrote: > Hi, > > Nathan, could you please advise what is expected startup time for OpenMPI > job at such scale (128K ranks)? I'm interesting in > 1) time from mpirun start to completion of MPI_Init()
It takes less than a minute to run: mpirun -n 131072 /bin/true > 2) time from MPI_Init() start to completion of MPI_Init() A simple MPI application took about about 1.25 mins to run. If you want to see our setup you can take a look at contrib/platform/lanl/cray_xe6. > >From my experience for 52800 rank job > 1) took around 20 min > 2) took around 12 min > that actually looks like a hung. How many nodes? I have never seen launch times that bad on Cielo. You could try adding -mca routed debruijn -novm and see if that helps. It will reduce the amount of communication between compute nodes and the login node. > Any advice how to improve startup times of large scale jobs would be very > much appreciated. The bottleneck for launching at scale is the initial communication between the orteds and mpirun. At this time I don't know what can be done to improve that (I have some ideas but nothing has been implemented yet). At 8192 nodes this takes less than a minute. Everything else should be fairly quick. -Nathan Hjelm HPC-3, LANL
