I think the next reasonable step is to use some kind of diagnostic to find out where and why the application is hung. padb is a great free/open source tool that can be used here.
On Apr 14, 2011, at 4:46 AM, Rushton Martin wrote: > I forwarded your question to the code custodian and received the > following reply (GRIM is the major code, the one which shows the > problem): "I've not tried the debugger but GRIM does have a number of > mpi_barrier calls in it so I would think we are safe there. There is of > course a performance downside with an over-use of barriers! As mentioned > in the e-trail." > > > Martin Rushton > HPC System Manager, Weapons Technologies > Tel: 01959 514777, Mobile: 07939 219057 > email: jmrush...@qinetiq.com > www.QinetiQ.com > QinetiQ - Delivering customer-focused solutions > > Please consider the environment before printing this email. > -----Original Message----- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of Ralph Castain > Sent: 14 April 2011 04:55 > To: Open MPI Users > Subject: Re: [OMPI users] Over committing? > > Have you folks used a debugger such as TotalView or padb to look at > these stalls? > > I ask because we discovered a long time ago that MPI collectives can > "hang" in the scenario you describe. It is caused by one rank falling > behind, and then never catching up due to resource allocations - i.e.., > once you fall behind due to the processor being used by something else, > you never catch up. > > The code that causes this is generally a loop around a collective such > as Allreduce. The solution was to inject a "barrier" operation in the > loop periodically, thus ensuring that all ranks had an opportunity to > catch up. > > There is an MCA param you can set that will inject the barrier - it > specifies to inject it every N collective operations (either before or > after the Nth op): > > -mca coll_sync_barrier_before N > > or > > -mca coll_sync_barrier_after N > > It'll slow the job down a little, depending upon how often you inject > the barrier. But it did allow us to run jobs reliably to completion when > the code involved such issues. > > > On Apr 13, 2011, at 10:07 AM, Rushton Martin wrote: > >> The 16 cores refers to x3755-m2s. We have a mix of 3550s and 3755s in > >> the cluster. >> >> It could be memory, but I think not. The jobs are well within memory >> capacity, and the memory is mainly static. If out of memory then the >> jobs would be first candidate for the job. Larger jobs run on the >> 3755s which as well as more memory have local disks for paging to. >> >> >> Martin Rushton >> HPC System Manager, Weapons Technologies >> Tel: 01959 514777, Mobile: 07939 219057 >> email: jmrush...@qinetiq.com >> www.QinetiQ.com >> QinetiQ - Delivering customer-focused solutions >> >> Please consider the environment before printing this email. >> -----Original Message----- >> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] >> On Behalf Of Reuti >> Sent: 13 April 2011 16:53 >> To: Open MPI Users >> Subject: Re: [OMPI users] Over committing? >> >> Am 13.04.2011 um 17:09 schrieb Rushton Martin: >> >>> Version 1.3.2 >>> >>> Consider a job that will run with 28 processes. The user submits it >>> with: >>> >>> $ qsub -l nodes=4:ppn=7 ... >>> >>> which reserves 7 cores on (in this case) each of x3550x014 x3550x015 >>> and >>> x3550x016 x3550x020. Torque generates a file (PBS_NODEFILE) which >>> lists each node 7 times. >>> >>> The mpirun command given within the batch script is: >>> >>> $ mpirun -np 28 -machinefile $PBS_NODEFILE <executable> >>> >>> This is what I would refer to as 7+7+7+7, and it runs fine. >>> >>> The problem occurs if, for instance, a 24 core job is attempted. >>> qsub >> >>> gets nodes=3:ppn=8 and mpirun has -np 24. The job is now running on >>> three nodes, using all eight cores on each node - 8+8+8. This sort >>> of >> >>> job will eventually hang and has to be killed off. >>> >>> Cores Nodes Ppn Result >>> ----- ----- --- ------ >>> 8 1 any works >>> 8 >1 1-7 works >>> 8 >1 8 hangs >>> 16 1 any works >>> 16 >1 1-15 works >>> 16 >1 16 hangs >> >> How many cores do you have in each system? Looks like 8 is the maximum > >> IBM offers from their datasheet, and still you can request 16 per > node? >> >> Can it be a memory porblem? >> >> -- Reuti >> >> >>> We have also tried test jobs on 8+7 (or 7+8) with inconclusive >> results. >>> Some of the live jobs run for a month or more and cut down versions >>> do >> >>> not model well. >>> >>> Martin Rushton >>> HPC System Manager, Weapons Technologies >>> Tel: 01959 514777, Mobile: 07939 219057 >>> email: jmrush...@qinetiq.com >>> www.QinetiQ.com >>> QinetiQ - Delivering customer-focused solutions >>> >>> Please consider the environment before printing this email. >>> -----Original Message----- >>> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] >>> On Behalf Of Ralph Castain >>> Sent: 13 April 2011 15:34 >>> To: Open MPI Users >>> Subject: Re: [OMPI users] Over committing? >>> >>> >>> On Apr 13, 2011, at 8:13 AM, Rushton Martin wrote: >>> >>>> The bulk of our compute nodes are 8 cores (twin 4-core IBM > x3550-m2). >>>> Jobs are submitted by Torque/MOAB. When run with up to np=8 there >>>> is >> >>>> good performance. Attempting to run with more processors brings >>>> problems, specifically if any one node of a group of nodes has all 8 > >>>> cores in use the job hangs. For instance running with 14 cores >>>> (7+7) >> >>>> is fine, but running with 16 (8+8) hangs. >>>> >>>>> From the FAQs I note the issues of over committing and aggressive >>>> scheduling. Is it possible for mpirun (or orted on the remote >>>> nodes) >> >>>> to be blocked from progressing by a fully committed node? We have a > >>>> few >>>> x3755-m2 machines with 16 cores, and we have detected a similar >>>> issue >> >>>> with 16+16. >>> >>> I'm not entirely sure I understand your notation, but we have never >>> seen an issue when running with fully loaded nodes (i.e., where the >>> number of MPI procs on the node = the number of cores). >>> >>> What version of OMPI are you using? Are you binding the procs? >>> This email and any attachments to it may be confidential and are >>> intended solely for the use of the individual to whom it is > addressed. >> >>> If you are not the intended recipient of this email, you must neither > >>> take any action based upon its contents, nor copy or show it to >>> anyone. Please contact the sender if you believe you have received >>> this email in error. QinetiQ may monitor email traffic data and also >>> the content of email for the purposes of security. QinetiQ Limited >>> (Registered in England & Wales: Company Number: 3796233) Registered >>> office: Cody Technology Park, Ively Road, Farnborough, Hampshire, >>> GU14 >> >>> 0LX http://www.qinetiq.com. >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> The QinetiQ e-mail privacy policy and company information is detailed > elsewhere in the body of this email. >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > The QinetiQ e-mail privacy policy and company information is detailed > elsewhere in the body of this email. > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
